diff --git a/.gitattributes b/.gitattributes index 8794fdc0a5c..45b42096b9d 100644 --- a/.gitattributes +++ b/.gitattributes @@ -7,3 +7,19 @@ /demos/build_demos_msvc.bat omz.package=w /demos/tests/** -omz.package /tools/downloader/tests/** -omz.package + +/ci/requirements-ac.txt omz.ci.job-for-change.ac +/ci/requirements-ac-test.txt omz.ci.job-for-change.ac +/ci/requirements-conversion.txt omz.ci.job-for-change.demos omz.ci.job-for-change.models +/ci/requirements-demos.txt omz.ci.job-for-change.demos +/ci/requirements-downloader.txt omz.ci.job-for-change.demos omz.ci.job-for-change.models + +/demos/** omz.ci.job-for-change.demos +/demos/**/*.md -omz.ci.job-for-change.demos + +/models/** omz.ci.job-for-change.documentation +/models/**/*.yml -omz.ci.job-for-change.documentation + +/tools/accuracy_checker/** omz.ci.job-for-change.ac +/tools/downloader/** omz.ci.job-for-change.downloader +/tools/**/*.md -omz.ci.job-for-change.ac -omz.ci.job-for-change.downloader diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000000..ec208adb8b6 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,276 @@ +# How to Contribute Models to Open Model Zoo + +We appreciate your intention to contribute model to the OpenVINO™ Open Model Zoo (OMZ). OMZ is licensed under the Apache\* License, Version 2.0. By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms. Note that we accept models under permissive licenses, such as **MIT**, **Apache 2.0**, and **BSD-3-Clause**. Otherwise, it might take longer time to get your model approved. + +Frameworks supported by the Open Model Zoo: +* Caffe\* +* Caffe2\* (via conversion to ONNX\*) +* TensorFlow\* +* PyTorch\* (via conversion to ONNX\*) +* MXNet\* + +Open Model Zoo also supports models already in the ONNX format. + +## Pull Request Requirements + +To contribute to OMZ, create a pull request (PR) in this repository using the `develop` branch. +Pull requests are strictly formalized and are reviewed by the OMZ maintainers for consistence and legal compliance. + +Each PR contributing a model must contain: +* [configuration file `model.yml`](#configuration-file) +* [documentation of model in markdown format](#documentation) +* [accuracy validation configuration file](#accuracy-validation) +* (*optional*) [demo](#demo) + +Follow the rules in the sections below before submitting a pull request. + +### Model Name + +Name your model in OMZ according to the following rules: +- Use a name that is consistent with an original name, but complete match is not necessary +- Use lowercase +- Use `-`(preferable) or `_` as delimiters, for spaces are not allowed +- Add a suffix according to framework identifier (see **`framework`** description in the [configuration file](#configuration-file) section for examples), if the model is a reimplementation of an existing model from another framework + +This name will be used for downloading, converting, and other operations. +Examples of model names: +- `resnet-50-pytorch` +- `mobilenet-v2-1.0-224` + +### Files Location + +Place your files as shown in the table below: + +File | Destination +---|--- +configuration file | `models/public//model.yml` +documentation file | `models/public//.md` +validation configuration file|`tools/accuracy_checker/configs/.yml` +demo|`demos/`
or
`demos/python_demos/` + +### Tests + +Your PR must pass next tests: +* Model is downloadable by the `tools/downloader/downloader.py` script. See [Configuration file](#configuration-file) for details. +* Model is convertible by the `tools/downloader/converter.py` script. See [Model conversion](#model-conversion) for details. +* Model is usable by demo or sample and provides adequate results. See [Demo](#demo) for details. +* Model passes accuracy validation. See [Accuracy validation](#accuracy-validation) for details. + + +### PR Rejection + +Your PR may be rejected in some cases, for example: +* If a license is inappropriate (such as GPL-like licenses). +* If a dataset is inaccessible. +* If the PR fails one of the tests above. + +## Configuration File + +The model configuration file contains information about model: what it is, how to download it, and how to convert it to the IR format. This information must be specified in the `model.yml` file that must be located in the model subfolder. + +The detailed descriptions of file entries provided below. + +**`description`** + +Description of the model. Must match with the description from the model [documentation](#documentation). + +**`task_type`** + +[Model task type](tools/downloader/README.md#model-information-dumper-usage). If there is no task type of your model, add a new one to the list `KNOWN_TASK_TYPES` of the [tools/downloader/common.py](tools/downloader/common.py) file. + +**`files`** + +> **NOTE**: Before filling this section, make sure that the model can be downloaded either via a direct HTTP(S) link or from Google Drive\*. + +Downloadable files. Each file is described by: + +* `name` - sets a file name after downloading +* `size` - sets a file size +* `sha256` - sets a file hash sum +* `source` - sets a direct link to a file *OR* describes a file access parameters + +> **TIP**: You can obtain a hash sum using the `sha256sum ` command on Linux\*. + +If file is located on Google Drive\*, the `source` section must contain: +- `$type: google_drive` +- `id` file ID on Google Drive\* + +> **NOTE:** If file is on GitHub\*, use the specific file version. + +**`postprocessing`** (*optional*) + +Post processing of the downloaded files. + +For unpacking archive: +- `$type: unpack_archive` +- `file` — Archive file name +- `format` — Archive format (zip | tar | gztar | bztar | xztar) + +For replacement operation: +- `$type: regex_replace` +- `file` — Name of file to run replacement in +- `pattern` — [Regular expression](https://docs.python.org/3/library/re.html) +- `replacement` — Replacement string +- `count` (*optional*) — Exact number of replacements (if number of `pattern` occurrences less then this number, downloading will be aborted) + +**`conversion_to_onnx_args`** (*only for Caffe2\*, PyTorch\* models*) + +List of ONNX\* conversion parameters, see `model_optimizer_args` for details. + +**`model_optimizer_args`** + +Conversion parameters (learn more in the [Model conversion](#model-conversion) section). For example: +``` + - --input=data + - --mean_values=data[127.5] + - --scale_values=data[127.5] + - --reverse_input_channels + - --output=prob + - --input_model=$conv_dir/googlenet-v3.onnx +``` + +> **NOTE:** Do not specify `framework`, `data_type`, `model_name` and `output_dir`, since they are deduced automatically. + +> **NOTE:** `$dl_dir` used to substitute subdirectory of downloaded model and `$conv_dir` used to substitute subdirectory of converted model. + +**`framework`** + +Framework of the original model (see [here](tools/downloader/README.md#model-information-dumper-usage) for details). + +**`license`** + +URL of the model license. + +### Example + +This example shows how to download the classification model [DenseNet-121*](models/public/densenet-121-tf/model.yml) pretrained in TensorFlow\* from Google Drive\* as an archive. + +``` +description: >- + This is a TensorFlow\* version of `densenet-121` model, one of the DenseNet + group of models designed to perform image classification. The weights were converted + from DenseNet-Keras Models. For details see repository , + paper +task_type: classification +files: + - name: tf-densenet121.tar.gz + size: 30597420 + sha256: b31ec840358f1d20e1c6364d05ce463cb0bc0480042e663ad54547189501852d + source: + $type: google_drive + id: 0B_fUSpodN0t0eW1sVk1aeWREaDA +postprocessing: + - $type: unpack_archive + format: gztar + file: tf-densenet121.tar.gz +model_optimizer_args: + - --reverse_input_channels + - --input_shape=[1,224,224,3] + - --input=Placeholder + - --mean_values=Placeholder[123.68,116.78,103.94] + - --scale_values=Placeholder[58.8235294117647] + - --output=densenet121/predictions/Reshape_1 + - --input_meta_graph=$dl_dir/tf-densenet121.ckpt.meta +framework: tf +license: https://raw.githubusercontent.com/pudae/tensorflow-densenet/master/LICENSE +``` + +## Model Conversion + +Deep Learning Inference Engine (IE) supports models in the Intermediate Representation (IR) format. A model from any supported framework can be converted to IR using the Model Optimizer tool included in the OpenVINO™ toolkit. Find more information about conversion in the [Model Optimizer Developer Guide](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html). After a successful conversion you get a model in the IR format, with the `*.xml` file representing the net graph and the `*.bin` file containing the net parameters. + +> **NOTE 1**: Image preprocessing parameters (mean and scale) must be built into a converted model to simplify model usage. + +> **NOTE 2**: If a model input is a color image, color channel order should be `BGR`. + +## Demo + +A demo shows the main idea of how to infer a model using IE. If your model solves one of the tasks supported by the Open Model Zoo, try to find an appropriate option from [demos](demos/README.md) or [samples](https://docs.openvinotoolkit.org/latest/_docs_IE_DG_Samples_Overview.html). Otherwise, you must provide your own demo (C++ or Python). + +The demo's name should end with `_demo` suffix to follow the convention of the project. + +Demos are required to support the following keys: + + - `-i ""`: Required. Input to process. + - `-m ""`: Required. Path to an .xml file with a trained model. If the demo uses several models at the same time, use other keys prefixed with `-m_`. + - `-d ""`: Optional. Specifies a target device to infer on. CPU, GPU, FPGA, HDDL or MYRIAD is acceptable. Default must be CPU. If the demo uses several models at the same time, use keys prefixed with `d_` (just like keys `m_*` above) to specify device for each model. + - `-no_show`: Optional. Do not visualize inference results. + +> **TIP**: For Python, it is preferable to use `--` instead of `-` for long keys. + +You can also add any other necessary parameters. + +Add `README.md` file, which describes demo usage. Update [demos' README.md](demos/README.md) adding your demo to the list. + +## Accuracy Validation + +Accuracy validation can be performed by the [Accuracy Checker](./tools/accuracy_checker) tool. This tool can use either IE to run a converted model, or an original framework to run an original model. Accuracy Checker supports lots of datasets, metrics and preprocessing options, which simplifies validation if a task is supported by the tool. You only need to create a configuration file that contains necessary parameters for accuracy validation (specify a dataset and annotation, pre- and post-processing parameters, accuracy metrics to compute and so on) of converted model. For details, refer to [Testing new models](./tools/accuracy_checker#testing-new-models). + +If a model uses a dataset which is not supported by the Accuracy Checker, you also must provide the license and the link to it and mention it in the PR description. + +When the configuration file is ready, you must run the Accuracy Checker to obtain metric results. If they match your results, that means conversion was successful and the Accuracy Checker fully supports your model, metric and dataset. Otherwise, recheck the [conversion](#model-conversion) parameters or the validation configuration file. + +### Example + +This example uses one of the files from `tools/accuracy_checker/configs` — validation configuration file for [DenseNet-121](tools/accuracy_checker/configs/densenet-121-tf.yml)\* from TensorFlow\*: +``` +models: + - name: densenet-121-tf + launchers: + - framework: dlsdk + tags: + - FP32 + model: public/densenet-121-tf/FP32/densenet-121-tf.xml + weights: public/densenet-121-tf/FP32/densenet-121-tf.bin + adapter: classification + + - framework: dlsdk + tags: + - FP16 + model: public/densenet-121-tf/FP16/densenet-121-tf.xml + weights: public/densenet-121-tf/FP16/densenet-121-tf.bin + adapter: classification + + datasets: + - name: imagenet_1000_classes + preprocessing: + - type: resize + size: 256 + - type: crop + size: 224 +``` + + +## Documentation + +Documentation is a very important part of model contribution as it helps to better understand the possible usage of the model. Documentation must be named in accordance with the name of the model. +The documentation should contain: +* description of a model + * main purpose + * features + * references to a paper or/and a source +* model specification + * type + * framework + * GFLOPs (*if available*) + * number of parameters (*if available*) +* validation dataset description and/or a link +* main accuracy values (also description of a metric) +* detailed description of input and output for original and converted models +* the model's licensing terms + +Learn the detailed structure and headers naming convention from any model documentation (for example, [alexnet](./models/public/alexnet/alexnet.md)). + +## Legal Information + +[\*] Other names and brands may be claimed as the property of others. + +OpenVINO is a trademark of Intel Corporation or its subsidiaries in the U.S. and/or other countries. + +Copyright © 2018-2019 Intel Corporation + +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at +``` +http://www.apache.org/licenses/LICENSE-2.0 +``` +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. diff --git a/README.md b/README.md index 1cd0e48b614..cfd71e8c23a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,5 @@ # [OpenVINO™ Toolkit](https://01.org/openvinotoolkit) - Open Model Zoo repository -[![Build Status](http://134.191.240.124/buildStatus/icon?job=omz/2018/trigger)](http://134.191.240.124/job/omz/job/2018/job/trigger/) -[![Stable release](https://img.shields.io/badge/version-2019_R3.1-green.svg)](https://github.com/opencv/open_model_zoo/releases/tag/2019_R3.1) +[![Stable release](https://img.shields.io/badge/version-2020.2-green.svg)](https://github.com/opencv/open_model_zoo/releases/tag/2020.2) [![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/open_model_zoo/community) [![Apache License Version 2.0](https://img.shields.io/badge/license-Apache_2.0-green.svg)](LICENSE) @@ -34,6 +33,8 @@ We welcome community contributions to the Open Model Zoo repository. If you have * In case of a larger feature, provide a relevant demo. * Submit a pull request at https://github.com/opencv/open_model_zoo/pulls +You can find additional information about model contribution [here](CONTRIBUTING.md). + We will review your contribution and, if any additional fixes or modifications are needed, may give you feedback to guide you. When accepted, your pull request will be merged into the GitHub* repositories. Open Model Zoo is licensed under Apache License, Version 2.0. By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms. diff --git a/ci/get-jobs-for-changes.py b/ci/get-jobs-for-changes.py new file mode 100755 index 00000000000..829a4f8f0a6 --- /dev/null +++ b/ci/get-jobs-for-changes.py @@ -0,0 +1,72 @@ +#!/usr/bin/python3 + +# Copyright (c) 2019 Intel Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A script that prints a JSON description of the CI jobs necessary to validate +the changes made between a base commit and the current commit. + +The output format is a an object where each key is the identifier of the job +and the corresponding value represents that job's parameters. The value +is usually just "true" (which just means that the job should be run), but +for the "models" job the value can be an array of names of models that should +be validated. +""" + +import argparse +import json +import re +import subprocess +import sys + +from pathlib import Path, PurePosixPath + +RE_ATTRIB_NAME = re.compile(r"omz\.ci\.job-for-change\.(.+)") + +def group_by_n(iterable, n): + return zip(*[iter(iterable)] * n) + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('base_commit', metavar='COMMIT') + args = parser.parse_args() + + git_diff_output = subprocess.check_output( + ["git", "diff", "--name-only", "-z", args.base_commit + "...HEAD"]) + changed_files = list(map(PurePosixPath, git_diff_output.decode()[:-1].split("\0"))) + + models_dir = PurePosixPath("models") + + jobs = {} + + for changed_file in changed_files: + if models_dir in changed_file.parents and changed_file.name == "model.yml": + if Path(changed_file).exists(): # it might've been deleted in the branch + jobs.setdefault("models", []).append(changed_file.parent.name) + + git_check_attr_output = subprocess.run( + ["git", "check-attr", "--stdin", "-z", "--all"], + input=git_diff_output, stdout=subprocess.PIPE, check=True).stdout + + for path, attribute, value in group_by_n(git_check_attr_output.decode()[:-1].split("\0"), 3): + attribute_match = RE_ATTRIB_NAME.fullmatch(attribute) + if value != 'unset' and attribute_match: + jobs[attribute_match.group(1)] = True + + json.dump(jobs, sys.stdout) + print() + +if __name__ == "__main__": + main() diff --git a/ci/requirements-ac-test.txt b/ci/requirements-ac-test.txt new file mode 100644 index 00000000000..3a066c4525d --- /dev/null +++ b/ci/requirements-ac-test.txt @@ -0,0 +1,31 @@ +# +# This file is autogenerated by pip-compile +# To update, run: +# +# pip-compile --output-file=ci/requirements-ac-test.txt tools/accuracy_checker/requirements-test.in tools/accuracy_checker/requirements.in +# +attrs==19.3.0 # via pytest +importlib-metadata==1.5.0 # via pluggy, pytest +joblib==0.14.1 # via scikit-learn +more-itertools==8.2.0 # via pytest +nibabel==3.0.1 +numpy==1.17.5 +packaging==20.1 # via pytest +pathlib2==2.3.5 # via pytest +pillow==7.0.0 +pluggy==0.13.1 # via pytest +py-cpuinfo==4.0.0 +py==1.8.1 # via pytest +pyparsing==2.4.6 # via packaging +pytest-mock==2.0.0 +pytest==5.3.5 +pyyaml==5.3 +scikit-learn==0.22.1 +scipy==1.4.1 +sentencepiece==0.1.85 +shapely==1.7.0 +six==1.14.0 # via packaging, pathlib2 +tqdm==4.42.1 +wcwidth==0.1.8 # via pytest +yamlloader==0.5.5 +zipp==1.1.0 # via importlib-metadata diff --git a/ci/requirements-ac.txt b/ci/requirements-ac.txt index 342f042f103..8fdbf925883 100644 --- a/ci/requirements-ac.txt +++ b/ci/requirements-ac.txt @@ -1,12 +1,18 @@ -joblib==0.13.2 # via scikit-learn -nibabel==2.5.0 -numpy==1.17.0 -pillow==6.1.0 +# +# This file is autogenerated by pip-compile +# To update, run: +# +# pip-compile --output-file=ci/requirements-ac.txt tools/accuracy_checker/requirements.in +# +joblib==0.14.1 # via scikit-learn +nibabel==3.0.1 +numpy==1.17.5 +pillow==7.0.0 py-cpuinfo==4.0.0 -pyyaml==5.1.2 -scikit-learn==0.21.3 -scipy==0.19.0 -shapely==1.6.4.post2 -six==1.12.0 # via nibabel -tqdm==4.33.0 +pyyaml==5.3 +scikit-learn==0.22.1 +scipy==1.4.1 +sentencepiece==0.1.85 +shapely==1.7.0 +tqdm==4.42.1 yamlloader==0.5.5 diff --git a/ci/requirements-conversion.txt b/ci/requirements-conversion.txt index d147fcefea7..d94e3079b24 100644 --- a/ci/requirements-conversion.txt +++ b/ci/requirements-conversion.txt @@ -1,40 +1,40 @@ -absl-py==0.7.1 # via tensorboard, tensorflow -astor==0.8.0 # via tensorflow -certifi==2019.6.16 # via requests +absl-py==0.9.0 # via tensorboard, tensorflow +astor==0.8.1 # via tensorflow +certifi==2019.11.28 # via requests chardet==3.0.4 # via requests -decorator==4.4.0 # via networkx +decorator==4.4.1 # via networkx defusedxml==0.6.0 +future==0.18.2 gast==0.2.2 # via tensorflow -google-pasta==0.1.7 # via tensorflow +google-pasta==0.1.8 # via tensorflow graphviz==0.8.4 # via mxnet -grpcio==1.22.0 # via tensorboard, tensorflow -h5py==2.9.0 # via keras-applications +grpcio==1.26.0 # via tensorboard, tensorflow +h5py==2.10.0 # via keras-applications idna==2.8 # via requests keras-applications==1.0.8 # via tensorflow keras-preprocessing==1.1.0 # via tensorflow markdown==3.1.1 # via tensorboard -mxnet==1.3.1 -networkx==2.3 -numpy==1.14.6 -onnx==1.5.0 -pillow==6.1.0 # via torchvision +mxnet==1.5.1 +networkx==2.4 +numpy==1.18.1 +onnx==1.6.0 +opt-einsum==3.1.0 # via tensorflow +pillow==7.0.0 # via torchvision protobuf==3.6.1 requests==2.22.0 # via mxnet -scipy==1.3.1 -six==1.12.0 # via absl-py, grpcio, h5py, keras-preprocessing, onnx, protobuf, tensorboard, tensorflow, test-generator, torchvision -tensorboard==1.14.0 # via tensorflow -tensorflow-estimator==1.14.0 # via tensorflow -tensorflow==1.14.0 +scipy==1.4.1 +six==1.14.0 # via absl-py, google-pasta, grpcio, h5py, keras-preprocessing, onnx, protobuf, tensorboard, tensorflow, torchvision +tensorboard==1.15.0 # via tensorflow +tensorflow-estimator==1.15.1 # via tensorflow +tensorflow==1.15.2 termcolor==1.1.0 # via tensorflow -test-generator==0.1.1 -torch==1.2.0 -torchvision==0.4.0 -typing-extensions==3.7.4 # via onnx -typing==3.7.4 # via onnx -urllib3==1.25.3 # via requests -werkzeug==0.15.5 # via tensorboard -wheel==0.33.4 # via tensorboard, tensorflow +torch==1.4.0 +torchvision==0.5.0 +typing-extensions==3.7.4.1 # via onnx +urllib3==1.25.8 # via requests +werkzeug==0.16.1 # via tensorboard +wheel==0.34.1 # via tensorboard, tensorflow wrapt==1.11.2 # via tensorflow # The following packages are considered to be unsafe in a requirements file: -# setuptools==41.0.1 # via markdown, protobuf, tensorboard +# setuptools diff --git a/ci/requirements-demos.txt b/ci/requirements-demos.txt index e07312c7697..b76e094ee0f 100644 --- a/ci/requirements-demos.txt +++ b/ci/requirements-demos.txt @@ -1 +1,6 @@ -numpy==1.17.0 ; python_version >= "3.4" +joblib==0.14.1 # via scikit-learn +nibabel==3.0.1 +numpy==1.18.1 ; python_version >= "3.4" +scikit-learn==0.22.1 +scipy==1.4.1 +tqdm==4.42.0 diff --git a/ci/requirements-downloader.txt b/ci/requirements-downloader.txt index 3f154ea4cb8..71005e1a89a 100644 --- a/ci/requirements-downloader.txt +++ b/ci/requirements-downloader.txt @@ -4,9 +4,9 @@ # # pip-compile --output-file=ci/requirements-downloader.txt tools/downloader/requirements.in # -certifi==2019.6.16 # via requests +certifi==2019.11.28 # via requests chardet==3.0.4 # via requests idna==2.8 # via requests -pyyaml==5.1.2 +pyyaml==5.3 requests==2.22.0 -urllib3==1.25.3 # via requests +urllib3==1.25.8 # via requests diff --git a/ci/update-requirements.py b/ci/update-requirements.py new file mode 100755 index 00000000000..4009758b4d0 --- /dev/null +++ b/ci/update-requirements.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 + +""" +This script updates all of the requirements-*.txt files in this directory +with the most recent package versions. + +It uses pip-compile (https://github.com/jazzband/pip-tools), so install that +before running it. +""" + +import os +import subprocess +import sys + +from pathlib import Path + +# Package dependencies can vary depending on the Python version. +# We thus have to run pip-compile with the lowest Python version that +# the project supports. +EXPECTED_PYTHON_VERSION = (3, 5) + +repo_root = Path(__file__).resolve().parent.parent + +def pip_compile(target, *sources): + print('updating {}...'.format(target), flush=True) + + # Use --no-header, since the OpenVINO install path may vary between machines, + # so it should not be embedded in the output file. Also, this script makes + # the information in pip-compile's headers redundant. + + subprocess.run( + [sys.executable, '-mpiptools', 'compile', + '--no-header', '--upgrade', '--quiet', '-o', target, '--', *map(str, sources)], + check=True, cwd=str(repo_root)) + +def main(): + if sys.version_info[:2] != EXPECTED_PYTHON_VERSION: + sys.exit("run this with Python {}".format('.'.join(map(str, EXPECTED_PYTHON_VERSION)))) + + if 'INTEL_OPENVINO_DIR' not in os.environ: + sys.exit("run OpenVINO toolkit's setupvars.sh before this") + + openvino_dir = Path(os.environ['INTEL_OPENVINO_DIR']) + + pip_compile('ci/requirements-ac.txt', + 'tools/accuracy_checker/requirements.in') + pip_compile('ci/requirements-ac-test.txt', + 'tools/accuracy_checker/requirements.in', 'tools/accuracy_checker/requirements-test.in') + pip_compile('ci/requirements-conversion.txt', + 'tools/downloader/requirements-pytorch.in', 'tools/downloader/requirements-caffe2.in', + openvino_dir / 'deployment_tools/model_optimizer/requirements.txt') + pip_compile('ci/requirements-demos.txt', + 'demos/python_demos/requirements.txt', openvino_dir / 'python/requirements.txt') + pip_compile('ci/requirements-downloader.txt', + 'tools/downloader/requirements.in') + +if __name__ == '__main__': + main() diff --git a/demos/CMakeLists.txt b/demos/CMakeLists.txt index ca6633fb808..376cf916d48 100644 --- a/demos/CMakeLists.txt +++ b/demos/CMakeLists.txt @@ -6,6 +6,8 @@ cmake_minimum_required (VERSION 2.8.12) project(Demos) +option(ENABLE_PYTHON "Whether to build extension modules for Python demos" OFF) + if (CMAKE_BUILD_TYPE STREQUAL "") message(STATUS "CMAKE_BUILD_TYPE not defined, 'Release' will be used") set(CMAKE_BUILD_TYPE "Release") @@ -97,7 +99,7 @@ if (${CMAKE_CXX_COMPILER_ID} STREQUAL GNU) set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall") endif() -add_subdirectory(common/format_reader) +add_subdirectory(common) # samples build can be switched off during whole IE build if (IE_MAIN_SOURCE_DIR AND NOT ENABLE_SAMPLES) @@ -155,10 +157,6 @@ macro(ie_add_sample) endif() endif() - if(TARGET IE::ie_cpu_extension) - add_definitions(-DWITH_EXTENSIONS) - endif() - # Create named folders for the sources within the .vcproj # Empty name lists them directly under the .vcproj source_group("src" FILES ${IE_SAMPLES_SOURCES}) @@ -178,10 +176,6 @@ macro(ie_add_sample) endif() target_include_directories(${IE_SAMPLE_NAME} PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/../common") - if(TARGET IE::ie_cpu_extension) - target_link_libraries(${IE_SAMPLE_NAME} PRIVATE IE::ie_cpu_extension) - endif() - target_link_libraries(${IE_SAMPLE_NAME} PRIVATE ${OpenCV_LIBRARIES} ${InferenceEngine_LIBRARIES} ${IE_SAMPLE_DEPENDENCIES} gflags) @@ -216,3 +210,10 @@ file(GLOB samples_dirs RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} *) # skip building of unnecessary subdirectories list(REMOVE_ITEM samples_dirs common thirdparty) add_samples_to_build(${samples_dirs}) + +if(ENABLE_PYTHON) + find_package(PythonInterp 3.5 REQUIRED) + find_package(PythonLibs "${PYTHON_VERSION_STRING}" EXACT REQUIRED) + + add_subdirectory(python_demos/human_pose_estimation_3d_demo/pose_extractor) +endif() diff --git a/demos/README.md b/demos/README.md index b41a62cc6da..21f8854e23e 100644 --- a/demos/README.md +++ b/demos/README.md @@ -5,7 +5,9 @@ The Open Model Zoo demo applications are console applications that demonstrate h The Open Model Zoo includes the following demos: +- [3D Human Pose Estimation Python* Demo](./python_demos/human_pose_estimation_3d_demo/README.md) - 3D human pose estimation demo. - [Action Recognition Python* Demo](./python_demos/action_recognition/README.md) - Demo application for Action Recognition algorithm, which classifies actions that are being performed on input video. +- [Colorization Python* Demo](./python_demos/colorization_demo/README.md) - Colorization demo colorizes input frames. - [Crossroad Camera C++ Demo](./crossroad_camera_demo/README.md) - Person Detection followed by the Person Attributes Recognition and Person Reidentification Retail, supports images/video and camera inputs. - [Gaze Estimation C++ Demo](./gaze_estimation_demo/README.md) - Face detection followed by gaze estimation, head pose estimation and facial landmarks regression. - [Human Pose Estimation C++ Demo](./human_pose_estimation_demo/README.md) - Human pose estimation demo. @@ -16,15 +18,18 @@ The Open Model Zoo includes the following demos: - [Interactive Face Recognition Python* Demo](./python_demos/face_recognition_demo/README.md) - Face Detection coupled with Head-Pose, Facial Landmarks and Face Recognition detectors. Supports video and camera inputs. - [Mask R-CNN C++ Demo for TensorFlow* Object Detection API](./mask_rcnn_demo/README.md) - Inference of instance segmentation networks created with TensorFlow\* Object Detection API. - [Multi-Camera Multi-Person Tracking Python* Demo](./python_demos/multi_camera_multi_person_tracking/README.md) Demo application for multiple persons tracking on multiple cameras. -- [Multi-Channel Face Detection C++ Demo](./multichannel_demo/README.md) - Simultaneous Multi Camera Face Detection demo. +- [Multi-Channel C++ Demos](./multi_channel/README.md) - Several demo applications for multi-channel scenarios. +- [Object Detection for CenterNet Python* Demo](./python_demos/object_detection_demo_centernet/README.md) - Demo application for CenterNet object detection network. - [Object Detection for Faster R-CNN C++ Demo](./object_detection_demo_faster_rcnn/README.md) - Inference of object detection networks like Faster R-CNN (the demo supports only images as inputs). - [Object Detection for SSD C++ Demo](./object_detection_demo_ssd_async/README.md) - Demo application for SSD-based Object Detection networks, new Async API performance showcase, and simple OpenCV interoperability (supports video and camera inputs). - [Object Detection for YOLO V3 C++ Demo](./object_detection_demo_yolov3_async/README.md) - Demo application for YOLOV3-based Object Detection networks, new Async API performance showcase, and simple OpenCV interoperability (supports video and camera inputs). - [Pedestrian Tracker C++ Demo](./pedestrian_tracker_demo/README.md) - Demo application for pedestrian tracking scenario. - [Security Barrier Camera C++ Demo](./security_barrier_camera_demo/README.md) - Vehicle Detection followed by the Vehicle Attributes and License-Plate Recognition, supports images/video and camera inputs. +- [Single Human Pose Estimation Python* Demo](./python_demos/single_human_pose_estimation_demo/README.md) - 2D human pose estimation demo. - [Smart Classroom C++ Demo](./smart_classroom_demo/README.md) - Face recognition and action detection demo for classroom environment. - [Super Resolution C++ Demo](./super_resolution_demo/README.md) - Super Resolution demo (the demo supports only images as inputs). It enhances the resolution of the input image. - [Text Detection C++ Demo](./text_detection_demo/README.md) - Text Detection demo. It detects and recognizes multi-oriented scene text on an input image and puts a bounding box around detected area. +- [Text Spotting Python* Demo](./python_demos/text_spotting_demo/README.md) - The demo demonstrates how to run Text Spotting models. \* Several C++ demos referenced above have simplified implementation in Python*, located in the `python_demos` directory. @@ -43,6 +48,7 @@ The table below shows the correlation between models, demos, and supported plugi | Model | Demos supported on the model | CPU | GPU | MYRIAD/HDDL | HETERO:FPGA,CPU | |--------------------------------------------------|----------------------------------------------------------------------------------------------------------------|-----------|-----------|-------------|-----------------| +| human-pose-estimation-3d-0001 | [3D Human Pose Estimation Python* Demo](./python_demos/human_pose_estimation_3d_demo/README.md) | Supported | Supported | | | | action-recognition-0001-decoder | [Action Recognition Demo](./python_demos/action_recognition/README.md) | Supported | Supported | | | | action-recognition-0001-encoder | [Action Recognition Demo](./python_demos/action_recognition/README.md) | Supported | Supported | | | | driver-action-recognition-adas-0002-decoder | [Action Recognition Demo](./python_demos/action_recognition/README.md) | Supported | Supported | | | @@ -68,10 +74,12 @@ The table below shows the correlation between models, demos, and supported plugi | license-plate-recognition-barrier-0001 | [Security Barrier Camera Demo](./security_barrier_camera_demo/README.md) | Supported | Supported | Supported | Supported | | vehicle-attributes-recognition-barrier-0039 | [Security Barrier Camera Demo](./security_barrier_camera_demo/README.md) | Supported | Supported | Supported | Supported | | vehicle-license-plate-detection-barrier-0106 | [Security Barrier Camera Demo](./security_barrier_camera_demo/README.md) | Supported | Supported | Supported | Supported | +| vehicle-license-plate-detection-barrier-0123 | [Security Barrier Camera Demo](./security_barrier_camera_demo/README.md) | Supported | Supported | Supported | Supported | | face-reidentification-retail-0095 | [Smart Classroom Demo](./smart_classroom_demo/README.md)
[Interactive Face Recognition Python* Demo](./python_demos/face_recognition_demo/README.md) | Supported | Supported | Supported | Supported | | landmarks-regression-retail-0009 | [Smart Classroom Demo](./smart_classroom_demo/README.md)
[Interactive Face Recognition Python* Demo](./python_demos/face_recognition_demo/README.md) | Supported | Supported | Supported | Supported | | person-detection-action-recognition-0005 | [Smart Classroom Demo](./smart_classroom_demo/README.md) | Supported | Supported | Supported | Supported | | person-detection-action-recognition-teacher-0002 | [Smart Classroom Demo](./smart_classroom_demo/README.md) | Supported | Supported | | Supported | +| single-human-pose-estimation-0001 | [Single Human Pose Estimation Python* Demo](./python_demos/single_human_pose_estimation_demo/README.md) | Supported | Supported | | single-image-super-resolution-1032 | [Super Resolution Demo](./super_resolution_demo/README.md) | Supported | Supported | | Supported | | single-image-super-resolution-1033 | [Super Resolution Demo](./super_resolution_demo/README.md) | Supported | Supported | | Supported | | text-detection-0003 | [Text Detection Demo](./text_detection_demo/README.md) | Supported | Supported | | Supported | @@ -146,7 +154,7 @@ make ``` For the release configuration, the demo application binaries are in `/intel64/Release/`; -for the debug configuration — in `/intel64/Debug/`. +for the debug configuration — in `/intel64/Debug/`. ### Build the Demos Applications on Microsoft Windows* OS @@ -155,18 +163,18 @@ The recommended Windows* build environment is the following: * Microsoft Visual Studio* 2015, 2017, or 2019 * CMake* version 2.8 or higher -> **NOTE**: If you want to use Microsoft Visual Studio 2019, you are required to install CMake 3.14. +> **NOTE**: If you want to use Microsoft Visual Studio 2019, you are required to install CMake 3.14. To build the demo applications for Windows, go to the directory with the `build_demos_msvc.bat` batch file and run it: -```sh +```bat build_demos_msvc.bat ``` By default, the script automatically detects the highest Microsoft Visual Studio version installed on the machine and uses it to create and build a solution for a demo code. Optionally, you can also specify the preffered Microsoft Visual Studio version to be used by the script. Supported versions are: `VS2015`, `VS2017`, `VS2019`. For example, to build the demos using the Microsoft Visual Studio 2017, use the following command: -```sh +```bat build_demos_msvc.bat VS2017 ``` @@ -177,6 +185,18 @@ build binaries in Debug configuration. Run the appropriate version of the Microsoft Visual Studio and open the generated solution file from the `C:\Users\\Documents\Intel\OpenVINO\omz_demos_build\Demos.sln` directory. +### Build the Native Python* Extension Modules + +Some of the Python demo applications require native Python extension modules to be built before they can be run. +This requires you to have Python development files (headers and import libraries) installed. +To build these modules, follow the instructions for building the demo applications above, +but add `-DENABLE_PYTHON=ON` to either the `cmake` or the `build_demos*` command, depending on which you use. +For example: + +```sh +cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_PYTHON=ON /demos +``` + ## Get Ready for Running the Demo Applications ### Get Ready for Running the Demo Applications on Linux* @@ -206,6 +226,14 @@ source /bin/setupvars.sh 3. Save and close the file: press the **Esc** key, type `:wq` and press the **Enter** key. 4. To test your change, open a new terminal. You will see `[setupvars.sh] OpenVINO environment initialized`. +To run Python demo applications that require native Python extension modules, you must additionally +set up the `PYTHONPATH` environment variable as follows, where `` is the directory with +the built demo applications: + +```sh +export PYTHONPATH="$PYTHONPATH:/lib" +``` + You are ready to run the demo applications. To learn about how to run a particular demo, read the demo documentation by clicking the demo name in the demo list above. @@ -215,12 +243,20 @@ list above. Before running compiled binary files, make sure your application can find the Inference Engine and OpenCV libraries. If you use a [proprietary](https://software.intel.com/en-us/openvino-toolkit) distribution to build demos, run the `setupvars` script to set all necessary environment variables: -```sh +```bat \bin\setupvars.bat ``` If you use your own Inference Engine and OpenCV binaries to build the demos please make sure you have added to the `PATH` environment variable. +To run Python demo applications that require native Python extension modules, you must additionally +set up the `PYTHONPATH` environment variable as follows, where `` is the directory with +the built demo applications: + +```bat +set PYTHONPATH=%PYTHONPATH%; +``` + To debug or run the demos on Windows in Microsoft Visual Studio, make sure you have properly configured **Debugging** environment settings for the **Debug** and **Release** configurations. Set correct paths to the OpenCV libraries, and @@ -229,7 +265,7 @@ For example, for the **Debug** configuration, go to the project's **Configuration Properties** to the **Debugging** category and set the `PATH` variable in the **Environment** field to the following: -```sh +``` PATH=\deployment_tools\inference_engine\bin\intel64\Debug;\opencv\bin;%PATH% ``` where `` is the directory in which the OpenVINO toolkit is installed. diff --git a/demos/build_demos.sh b/demos/build_demos.sh index 60e7d79cf0c..e13f4e03046 100755 --- a/demos/build_demos.sh +++ b/demos/build_demos.sh @@ -25,6 +25,20 @@ error() { } trap 'error ${LINENO}' ERR +extra_cmake_opts=() + +for opt in "$@"; do + case "$opt" in + -DENABLE_PYTHON=*) + extra_cmake_opts+=("$opt") + ;; + *) + printf "Unknown option: %q\n" "$opt" + exit 1 + ;; + esac +done + DEMOS_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" printf "\nSetting environment variables for building demos...\n" @@ -38,10 +52,10 @@ if [ -z "$INTEL_OPENVINO_DIR" ]; then printf "Error: Failed to set the environment variables automatically. To fix, run the following command:\n source /bin/setupvars.sh\n where INSTALL_DIR is the OpenVINO installation directory.\n\n" exit 1 fi - if ! source $setvars_path ; then + if ! source "$setvars_path"; then printf "Unable to run ./setupvars.sh. Please check its presence. \n\n" - exit 1 -fi + exit 1 + fi else # case for run with `sudo -E` source "$INTEL_OPENVINO_DIR/bin/setupvars.sh" @@ -57,17 +71,17 @@ build_dir=$HOME/omz_demos_build OS_PATH=$(uname -m) NUM_THREADS="-j2" -if [ $OS_PATH == "x86_64" ]; then - OS_PATH="intel64" - NUM_THREADS="-j8" +if [ "$OS_PATH" == "x86_64" ]; then + OS_PATH="intel64" + NUM_THREADS="-j8" fi -if [ -e $build_dir/CMakeCache.txt ]; then - rm -rf $build_dir/CMakeCache.txt +if [ -e "$build_dir/CMakeCache.txt" ]; then + rm -rf "$build_dir/CMakeCache.txt" fi -mkdir -p $build_dir -cd $build_dir -cmake -DCMAKE_BUILD_TYPE=Release $DEMOS_PATH -make $NUM_THREADS +mkdir -p "$build_dir" + +(cd "$build_dir" && cmake -DCMAKE_BUILD_TYPE=Release "${extra_cmake_opts[@]}" "$DEMOS_PATH") +cmake --build "$build_dir" -- "$NUM_THREADS" -printf "\nBuild completed, you can find binaries for all demos in the $build_dir/${OS_PATH}/Release subfolder.\n\n" +printf "\nBuild completed, you can find binaries for all demos in the %s subfolder.\n\n" "$build_dir/$OS_PATH/Release" diff --git a/demos/build_demos_msvc.bat b/demos/build_demos_msvc.bat index cc8728c9d61..6eb9737fdc6 100644 --- a/demos/build_demos_msvc.bat +++ b/demos/build_demos_msvc.bat @@ -15,8 +15,7 @@ :: limitations under the License. -@setlocal -SETLOCAL EnableDelayedExpansion +setlocal EnableDelayedExpansion set "ROOT_DIR=%~dp0" set "SOLUTION_DIR64=%USERPROFILE%\Documents\Intel\OpenVINO\omz_demos_build" @@ -24,122 +23,141 @@ set "SOLUTION_DIR64=%USERPROFILE%\Documents\Intel\OpenVINO\omz_demos_build" set MSBUILD_BIN= set VS_PATH= set VS_VERSION= +set EXTRA_CMAKE_OPTS= +:argParse if not "%1" == "" ( - if "%1"=="VS2015" ( - set "VS_VERSION=2015" - ) else if "%1"=="VS2017" ( - set "VS_VERSION=2017" - ) else if "%1"=="VS2019" ( - set "VS_VERSION=2019" - ) else ( - echo Unrecognized option specified "%1" - echo Supported command line options: VS2015, VS2017, VS2019 - goto errorHandling - ) + rem cmd.exe mangles -DENABLE_PYTHON=YES into -DENABLE_PYTHON YES, + rem so it gets split into two arguments + if "%1" == "-DENABLE_PYTHON" ( + set EXTRA_CMAKE_OPTS=%EXTRA_CMAKE_OPTS% %1=%2 + shift & shift + goto argParse + ) + + if not "%VS_VERSION%" == "" ( + echo Unexpected argument: "%1" + goto errorHandling + ) + + if "%1"=="VS2015" ( + set "VS_VERSION=2015" + ) else if "%1"=="VS2017" ( + set "VS_VERSION=2017" + ) else if "%1"=="VS2019" ( + set "VS_VERSION=2019" + ) else ( + echo Unrecognized Visual Studio version specified: "%1" + echo Supported versions: VS2015, VS2017, VS2019 + goto errorHandling + ) + + shift + goto argparse ) if "%INTEL_OPENVINO_DIR%"=="" ( if exist "%ROOT_DIR%\..\..\bin\setupvars.bat" ( call "%ROOT_DIR%\..\..\bin\setupvars.bat" + ) else if exist "%ROOT_DIR%\..\..\..\bin\setupvars.bat" ( + call "%ROOT_DIR%\..\..\..\bin\setupvars.bat" ) else ( - if exist "%ROOT_DIR%\..\..\..\bin\setupvars.bat" ( - call "%ROOT_DIR%\..\..\..\bin\setupvars.bat" - ) else ( - echo Failed to set the environment variables automatically - echo To fix, run the following command: ^\bin\setupvars.bat - echo where INSTALL_DIR is the OpenVINO installation directory. - GOTO errorHandling - ) + echo Failed to set the environment variables automatically + echo To fix, run the following command: ^\bin\setupvars.bat + echo where INSTALL_DIR is the OpenVINO installation directory. + goto errorHandling ) -) +) if "%PROCESSOR_ARCHITECTURE%" == "AMD64" ( - set "PLATFORM=x64" + set "PLATFORM=x64" ) else ( - set "PLATFORM=Win32" + set "PLATFORM=Win32" ) set VSWHERE="false" if exist "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" ( - set VSWHERE="true" - cd "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer" + set VSWHERE="true" + cd "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer" ) else if exist "%ProgramFiles%\Microsoft Visual Studio\Installer\vswhere.exe" ( - set VSWHERE="true" - cd "%ProgramFiles%\Microsoft Visual Studio\Installer" + set VSWHERE="true" + cd "%ProgramFiles%\Microsoft Visual Studio\Installer" ) else ( - echo "vswhere tool is not found" + echo "vswhere tool is not found" ) if !VSWHERE! == "true" ( - if "!VS_VERSION!"=="" ( - echo Searching the latest Visual Studio... - for /f "usebackq tokens=*" %%i in (`vswhere -latest -products * -requires Microsoft.Component.MSBuild -property installationPath`) do ( - set VS_PATH=%%i - ) - ) else ( - echo Searching Visual Studio !VS_VERSION!... - for /f "usebackq tokens=*" %%i in (`vswhere -products * -requires Microsoft.Component.MSBuild -property installationPath`) do ( - set CUR_VS=%%i - if not "!CUR_VS:%VS_VERSION%=!"=="!CUR_VS!" ( - set VS_PATH=!CUR_VS! - ) - ) - ) - if exist "!VS_PATH!\MSBuild\14.0\Bin\MSBuild.exe" ( - set "MSBUILD_BIN=!VS_PATH!\MSBuild\14.0\Bin\MSBuild.exe" - ) - if exist "!VS_PATH!\MSBuild\15.0\Bin\MSBuild.exe" ( - set "MSBUILD_BIN=!VS_PATH!\MSBuild\15.0\Bin\MSBuild.exe" - ) - if exist "!VS_PATH!\MSBuild\Current\Bin\MSBuild.exe" ( - set "MSBUILD_BIN=!VS_PATH!\MSBuild\Current\Bin\MSBuild.exe" - ) + if "!VS_VERSION!"=="" ( + echo Searching the latest Visual Studio... + for /f "usebackq tokens=*" %%i in (`vswhere -latest -products * -requires Microsoft.Component.MSBuild -property installationPath`) do ( + set VS_PATH=%%i + ) + ) else ( + echo Searching Visual Studio !VS_VERSION!... + for /f "usebackq tokens=*" %%i in (`vswhere -products * -requires Microsoft.Component.MSBuild -property installationPath`) do ( + set CUR_VS=%%i + if not "!CUR_VS:%VS_VERSION%=!"=="!CUR_VS!" ( + set VS_PATH=!CUR_VS! + ) + ) + ) + if exist "!VS_PATH!\MSBuild\14.0\Bin\MSBuild.exe" ( + set "MSBUILD_BIN=!VS_PATH!\MSBuild\14.0\Bin\MSBuild.exe" + ) + if exist "!VS_PATH!\MSBuild\15.0\Bin\MSBuild.exe" ( + set "MSBUILD_BIN=!VS_PATH!\MSBuild\15.0\Bin\MSBuild.exe" + ) + if exist "!VS_PATH!\MSBuild\Current\Bin\MSBuild.exe" ( + set "MSBUILD_BIN=!VS_PATH!\MSBuild\Current\Bin\MSBuild.exe" + ) ) if "!MSBUILD_BIN!" == "" ( - if "!VS_VERSION!"=="2015" ( - if exist "C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" ( - set "MSBUILD_BIN=C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" - set "MSBUILD_VERSION=14 2015" - ) - ) else if "!VS_VERSION!"=="2017" ( - if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" ( - set "MSBUILD_BIN=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" - set "MSBUILD_VERSION=15 2017" - ) else if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\MSBuild.exe" ( - set "MSBUILD_BIN=C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\MSBuild.exe" - set "MSBUILD_VERSION=15 2017" - ) else if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe" ( - set "MSBUILD_BIN=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe" - set "MSBUILD_VERSION=15 2017" - ) - ) + if "!VS_VERSION!"=="2015" ( + if exist "C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" ( + set "MSBUILD_BIN=C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" + set "MSBUILD_VERSION=14 2015" + ) + ) else if "!VS_VERSION!"=="2017" ( + if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" ( + set "MSBUILD_BIN=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" + set "MSBUILD_VERSION=15 2017" + ) else if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\MSBuild.exe" ( + set "MSBUILD_BIN=C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\MSBuild.exe" + set "MSBUILD_VERSION=15 2017" + ) else if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe" ( + set "MSBUILD_BIN=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe" + set "MSBUILD_VERSION=15 2017" + ) + ) ) else ( - if not "!MSBUILD_BIN:2019=!"=="!MSBUILD_BIN!" set "MSBUILD_VERSION=16 2019" - if not "!MSBUILD_BIN:2017=!"=="!MSBUILD_BIN!" set "MSBUILD_VERSION=15 2017" - if not "!MSBUILD_BIN:2015=!"=="!MSBUILD_BIN!" set "MSBUILD_VERSION=14 2015" + if not "!MSBUILD_BIN:2019=!"=="!MSBUILD_BIN!" set "MSBUILD_VERSION=16 2019" + if not "!MSBUILD_BIN:2017=!"=="!MSBUILD_BIN!" set "MSBUILD_VERSION=15 2017" + if not "!MSBUILD_BIN:2015=!"=="!MSBUILD_BIN!" set "MSBUILD_VERSION=14 2015" ) if "!MSBUILD_BIN!" == "" ( - echo Build tools for Microsoft Visual Studio !VS_VERSION! cannot be found. If you use Visual Studio 2017, please download and install build tools from https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017 - GOTO errorHandling + echo Build tools for Microsoft Visual Studio !VS_VERSION! cannot be found. If you use Visual Studio 2017, please download and install build tools from https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017 + goto errorHandling ) if exist "%SOLUTION_DIR64%\CMakeCache.txt" del "%SOLUTION_DIR64%\CMakeCache.txt" echo Creating Visual Studio %MSBUILD_VERSION% %PLATFORM% files in %SOLUTION_DIR64%... && ^ -cd "%ROOT_DIR%" && cmake -E make_directory "%SOLUTION_DIR64%" && cd "%SOLUTION_DIR64%" && cmake -G "Visual Studio !MSBUILD_VERSION!" -A %PLATFORM% "%ROOT_DIR%" +cd "%ROOT_DIR%" && cmake -E make_directory "%SOLUTION_DIR64%" + +cd "%SOLUTION_DIR64%" && cmake -G "Visual Studio !MSBUILD_VERSION!" -A %PLATFORM% %EXTRA_CMAKE_OPTS% "%ROOT_DIR%" echo. echo ###############^|^| Build Inference Engine Demos using MS Visual Studio (MSBuild.exe) ^|^|############### echo. echo "!MSBUILD_BIN!" Demos.sln /p:Configuration=Release "!MSBUILD_BIN!" Demos.sln /p:Configuration=Release -if ERRORLEVEL 1 GOTO errorHandling +if ERRORLEVEL 1 goto errorHandling echo Done. goto :eof :errorHandling echo Error +exit /B 1 diff --git a/demos/common/CMakeLists.txt b/demos/common/CMakeLists.txt new file mode 100644 index 00000000000..20f29f9b051 --- /dev/null +++ b/demos/common/CMakeLists.txt @@ -0,0 +1,5 @@ +# Copyright (C) 2018-2019 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# + +add_subdirectory(monitors) diff --git a/demos/common/format_reader/CMakeLists.txt b/demos/common/format_reader/CMakeLists.txt deleted file mode 100644 index c4011c48a70..00000000000 --- a/demos/common/format_reader/CMakeLists.txt +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (C) 2018-2019 Intel Corporation -# SPDX-License-Identifier: Apache-2.0 -# - -set (TARGET_NAME "format_reader") - -file (GLOB MAIN_SRC - ${CMAKE_CURRENT_SOURCE_DIR}/*.cpp - ) - -file (GLOB LIBRARY_HEADERS - ${CMAKE_CURRENT_SOURCE_DIR}/*.h - ) - -# Find OpenCV components if exist -find_package(OpenCV COMPONENTS imgcodecs videoio imgproc QUIET) -if(NOT(OpenCV_FOUND)) - message(WARNING "OPENCV is disabled or not found, " ${TARGET_NAME} " is built without OPENCV support") -endif() - -# Create named folders for the sources within the .vcproj -# Empty name lists them directly under the .vcproj -source_group("src" FILES ${LIBRARY_SRC}) -source_group("include" FILES ${LIBRARY_HEADERS}) - - -# Create library file from sources. -add_library(${TARGET_NAME} SHARED ${MAIN_SRC} ${LIBRARY_HEADERS}) - -if(OpenCV_FOUND) - target_link_libraries(${TARGET_NAME} PRIVATE ${OpenCV_LIBRARIES}) - target_compile_definitions(${TARGET_NAME} PRIVATE USE_OPENCV) -endif() - -target_compile_definitions(${TARGET_NAME} PRIVATE IMPLEMENT_FORMAT_READER) - -target_include_directories(${TARGET_NAME} PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}" - "${CMAKE_CURRENT_SOURCE_DIR}/..") - -set_target_properties(${TARGET_NAME} PROPERTIES COMPILE_PDB_NAME ${TARGET_NAME}) - -# developer package - -export(TARGETS ${TARGET_NAME} NAMESPACE IE:: - APPEND FILE "${CMAKE_BINARY_DIR}/targets_developer.cmake") diff --git a/demos/common/format_reader/MnistUbyte.cpp b/demos/common/format_reader/MnistUbyte.cpp deleted file mode 100644 index 6e46f0ec7b7..00000000000 --- a/demos/common/format_reader/MnistUbyte.cpp +++ /dev/null @@ -1,63 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#include -#include -#include -#include - -using namespace FormatReader; - -int MnistUbyte::reverseInt(int i) { - unsigned char ch1, ch2, ch3, ch4; - ch1 = (unsigned char) (i & 255); - ch2 = (unsigned char) ((i >> 8) & 255); - ch3 = (unsigned char) ((i >> 16) & 255); - ch4 = (unsigned char) ((i >> 24) & 255); - return (static_cast(ch1) << 24) + (static_cast(ch2) << 16) + (static_cast(ch3) << 8) + ch4; -} - -MnistUbyte::MnistUbyte(const std::string &filename) { - std::ifstream file(filename, std::ios::binary); - if (!file.is_open()) { - return; - } - int magic_number = 0; - int number_of_images = 0; - int n_rows = 0; - int n_cols = 0; - file.read(reinterpret_cast(&magic_number), sizeof(magic_number)); - magic_number = reverseInt(magic_number); - if (magic_number != 2051) { - return; - } - file.read(reinterpret_cast(&number_of_images), sizeof(number_of_images)); - number_of_images = reverseInt(number_of_images); - file.read(reinterpret_cast(&n_rows), sizeof(n_rows)); - n_rows = reverseInt(n_rows); - _height = (size_t) n_rows; - file.read(reinterpret_cast(&n_cols), sizeof(n_cols)); - n_cols = reverseInt(n_cols); - _width = (size_t) n_cols; - if (number_of_images > 1) { - std::cout << "[MNIST] Warning: number_of_images in mnist file equals " << number_of_images - << ". Only a first image will be read." << std::endl; - } - - size_t size = _width * _height * 1; - - _data.reset(new unsigned char[size], std::default_delete()); - size_t count = 0; - if (0 < number_of_images) { - for (int r = 0; r < n_rows; ++r) { - for (int c = 0; c < n_cols; ++c) { - unsigned char temp = 0; - file.read(reinterpret_cast(&temp), sizeof(temp)); - _data.get()[count++] = temp; - } - } - } - - file.close(); -} diff --git a/demos/common/format_reader/MnistUbyte.h b/demos/common/format_reader/MnistUbyte.h deleted file mode 100644 index fd6ae0f75e8..00000000000 --- a/demos/common/format_reader/MnistUbyte.h +++ /dev/null @@ -1,58 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -/** - * \brief Mnist reader - * \file MnistUbyte.h - */ -#pragma once - -#include -#include -#include - -#include "register.h" - -namespace FormatReader { -/** - * \class MnistUbyte - * \brief Reader for mnist db files - */ -class MnistUbyte : public Reader { -private: - int reverseInt(int i); - - static Register reg; - -public: - /** - * \brief Constructor of Mnist reader - * @param filename - path to input data - * @return MnistUbyte reader object - */ - explicit MnistUbyte(const std::string &filename); - virtual ~MnistUbyte() { - } - - /** - * \brief Get size - * @return size - */ - size_t size() const override { - return _width * _height * 1; - } - - void Release() noexcept override { - delete this; - } - - std::shared_ptr getData(size_t width, size_t height) override { - if ((width * height != 0) && (_width * _height != width * height)) { - std::cout << "[ WARNING ] Image won't be resized! Please use OpenCV.\n"; - return nullptr; - } - return _data; - } -}; -} // namespace FormatReader diff --git a/demos/common/format_reader/bmp.cpp b/demos/common/format_reader/bmp.cpp deleted file mode 100644 index b52f839abcb..00000000000 --- a/demos/common/format_reader/bmp.cpp +++ /dev/null @@ -1,63 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#include "bmp.h" -#include -#include - -using namespace std; -using namespace FormatReader; - - -BitMap::BitMap(const string &filename) { - BmpHeader header; - BmpInfoHeader infoHeader; - - ifstream input(filename, ios::binary); - if (!input) { - return; - } - - input.read(reinterpret_cast(&header.type), 2); - - if (header.type != 'M'*256+'B') { - std::cerr << "[BMP] file is not bmp type\n"; - return; - } - - input.read(reinterpret_cast(&header.size), 4); - input.read(reinterpret_cast(&header.reserved), 4); - input.read(reinterpret_cast(&header.offset), 4); - - input.read(reinterpret_cast(&infoHeader), sizeof(BmpInfoHeader)); - - - bool rowsReversed = infoHeader.height < 0; - _width = infoHeader.width; - _height = abs(infoHeader.height); - - if (infoHeader.bits != 24) { - cerr << "[BMP] 24bpp only supported. But input has:" << infoHeader.bits << "\n"; - return; - } - - if (infoHeader.compression != 0) { - cerr << "[BMP] compression not supported\n"; - } - - int padSize = _width & 3; - char pad[3]; - size_t size = _width * _height * 3; - - _data.reset(new unsigned char[size], std::default_delete()); - - input.seekg(header.offset, ios::beg); - - // reading by rows in invert vertically - for (uint32_t i = 0; i < _height; i++) { - uint32_t storeAt = rowsReversed ? i : (uint32_t)_height - 1 - i; - input.read(reinterpret_cast(_data.get()) + _width * 3 * storeAt, _width * 3); - input.read(pad, padSize); - } -} diff --git a/demos/common/format_reader/bmp.h b/demos/common/format_reader/bmp.h deleted file mode 100644 index b1b05dfc249..00000000000 --- a/demos/common/format_reader/bmp.h +++ /dev/null @@ -1,75 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -/** - * \brief BMP reader - * \file bmp.h - */ -#pragma once - -#include -#include -#include - -#include "register.h" - -namespace FormatReader { -/** - * \class BitMap - * \brief Reader for bmp files - */ -class BitMap : public Reader { -private: - static Register reg; - - typedef struct { - unsigned short type = 0u; /* Magic identifier */ - unsigned int size = 0u; /* File size in bytes */ - unsigned int reserved = 0u; - unsigned int offset = 0u; /* Offset to image data, bytes */ - } BmpHeader; - - typedef struct { - unsigned int size = 0u; /* Header size in bytes */ - int width = 0, height = 0; /* Width and height of image */ - unsigned short planes = 0u; /* Number of colour planes */ - unsigned short bits = 0u; /* Bits per pixel */ - unsigned int compression = 0u; /* Compression type */ - unsigned int imagesize = 0u; /* Image size in bytes */ - int xresolution = 0, yresolution = 0; /* Pixels per meter */ - unsigned int ncolours = 0u; /* Number of colours */ - unsigned int importantcolours = 0u; /* Important colours */ - } BmpInfoHeader; - -public: - /** - * \brief Constructor of BMP reader - * @param filename - path to input data - * @return BitMap reader object - */ - explicit BitMap(const std::string &filename); - virtual ~BitMap() { - } - - /** - * \brief Get size - * @return size - */ - size_t size() const override { - return _width * _height * 3; - } - - void Release() noexcept override { - delete this; - } - - std::shared_ptr getData(size_t width, size_t height) override { - if ((width * height != 0) && (_width * _height != width * height)) { - std::cout << "[ WARNING ] Image won't be resized! Please use OpenCV.\n"; - return nullptr; - } - return _data; - } -}; -} // namespace FormatReader diff --git a/demos/common/format_reader/format_reader.cpp b/demos/common/format_reader/format_reader.cpp deleted file mode 100644 index 30f33453216..00000000000 --- a/demos/common/format_reader/format_reader.cpp +++ /dev/null @@ -1,37 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#include -#include -#include "bmp.h" -#include "MnistUbyte.h" -#include "opencv_wraper.h" - -using namespace FormatReader; - -std::vector Registry::_data; - -Register MnistUbyte::reg; -#ifdef USE_OPENCV -Register OCVReader::reg; -#else -Register BitMap::reg; -#endif - -Reader *Registry::CreateReader(const char *filename) { - for (auto maker : _data) { - Reader *ol = maker(filename); - if (ol != nullptr && ol->size() != 0) return ol; - if (ol != nullptr) ol->Release(); - } - return nullptr; -} - -void Registry::RegisterReader(CreatorFunction f) { - _data.push_back(f); -} - -FORMAT_READER_API(Reader*) CreateFormatReader(const char *filename) { - return Registry::CreateReader(filename); -} \ No newline at end of file diff --git a/demos/common/format_reader/format_reader.h b/demos/common/format_reader/format_reader.h deleted file mode 100644 index d0c746275b7..00000000000 --- a/demos/common/format_reader/format_reader.h +++ /dev/null @@ -1,81 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -/** - * \brief Format reader abstract class implementation - * \file format_reader.h - */ -#pragma once - -#include -#include -#include -#include - -#if defined(_WIN32) -# ifdef IMPLEMENT_FORMAT_READER -# define FORMAT_READER_API(type) extern "C" __declspec(dllexport) type -# else -# define FORMAT_READER_API(type) extern "C" type -# endif -#elif(__GNUC__ >= 4) -# ifdef IMPLEMENT_FORMAT_READER -# define FORMAT_READER_API(type) extern "C" __attribute__((visibility("default"))) type -# else -# define FORMAT_READER_API(type) extern "C" type -# endif -#else -# define FORMAT_READER_API(TYPE) extern "C" TYPE -#endif - - -namespace FormatReader { -/** - * \class FormatReader - * \brief This is an abstract class for reading input data - */ -class Reader { -protected: - /// \brief height - size_t _height = 0; - /// \brief width - size_t _width = 0; - /// \brief data - std::shared_ptr _data; - -public: - /** - * \brief Get width - * @return width - */ - size_t width() const { return _width; } - - /** - * \brief Get height - * @return height - */ - size_t height() const { return _height; } - - /** - * \brief Get input data ptr - * @return shared pointer with input data - * @In case of using OpenCV, parameters width and height will be used for image resizing - */ - virtual std::shared_ptr getData(size_t width = 0, size_t height = 0) = 0; - - /** - * \brief Get size - * @return size - */ - virtual size_t size() const = 0; - - virtual void Release() noexcept = 0; -}; -} // namespace FormatReader - -/** - * \brief Function for create reader - * @return FormatReader pointer - */ -FORMAT_READER_API(FormatReader::Reader*) CreateFormatReader(const char *filename); \ No newline at end of file diff --git a/demos/common/format_reader/format_reader_ptr.h b/demos/common/format_reader/format_reader_ptr.h deleted file mode 100644 index 0b82d467457..00000000000 --- a/demos/common/format_reader/format_reader_ptr.h +++ /dev/null @@ -1,45 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -/** - * \brief Implementation of smart pointer for Reader class - * \file format_reader_ptr.h - */ -#pragma once - -#include "format_reader.h" -#include -#include - -namespace FormatReader { -class ReaderPtr { -public: - explicit ReaderPtr(const char *imageName) : reader(CreateFormatReader(imageName), - [](Reader *p) { - p->Release(); - }) {} - /** - * @brief dereference operator overload - * @return Reader - */ - Reader *operator->() const noexcept { - return reader.get(); - } - - /** - * @brief dereference operator overload - * @return Reader - */ - Reader *operator*() const noexcept { - return reader.get(); - } - - Reader *get() { - return reader.get(); - } - -protected: - std::unique_ptr> reader; -}; -} // namespace FormatReader diff --git a/demos/common/format_reader/opencv_wraper.cpp b/demos/common/format_reader/opencv_wraper.cpp deleted file mode 100644 index 835402ab6a7..00000000000 --- a/demos/common/format_reader/opencv_wraper.cpp +++ /dev/null @@ -1,48 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#ifdef USE_OPENCV -#include "opencv_wraper.h" -#include -#include - -#include - -#include - -using namespace std; -using namespace FormatReader; - -OCVReader::OCVReader(const string &filename) { - img = cv::imread(filename); - _size = 0; - - if (img.empty()) { - return; - } - - _size = img.size().width * img.size().height * img.channels(); - _width = img.size().width; - _height = img.size().height; -} - -std::shared_ptr OCVReader::getData(size_t width = 0, size_t height = 0) { - cv::Mat resized(img); - if (width != 0 && height != 0) { - size_t iw = img.size().width; - size_t ih = img.size().height; - if (width != iw || height != ih) { - slog::warn << "Image is resized from (" << iw << ", " << ih << ") to (" << width << ", " << height << ")" << slog::endl; - } - cv::resize(img, resized, cv::Size(width, height)); - } - - size_t size = resized.size().width * resized.size().height * resized.channels(); - _data.reset(new unsigned char[size], std::default_delete()); - for (size_t id = 0; id < size; ++id) { - _data.get()[id] = resized.data[id]; - } - return _data; -} -#endif diff --git a/demos/common/format_reader/opencv_wraper.h b/demos/common/format_reader/opencv_wraper.h deleted file mode 100644 index 5dc0b12f4da..00000000000 --- a/demos/common/format_reader/opencv_wraper.h +++ /dev/null @@ -1,56 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -/** - * \brief Image reader - * \file opencv_wraper.h - */ -#pragma once - -#ifdef USE_OPENCV -#include -#include -#include - -#include - -#include "register.h" - -namespace FormatReader { -/** - * \class OCVMAT - * \brief OpenCV Wraper - */ -class OCVReader : public Reader { -private: - cv::Mat img; - size_t _size; - static Register reg; - -public: - /** - * \brief Constructor of BMP reader - * @param filename - path to input data - * @return BitMap reader object - */ - explicit OCVReader(const std::string &filename); - virtual ~OCVReader() { - } - - /** - * \brief Get size - * @return size - */ - size_t size() const override { - return _size; - } - - void Release() noexcept override { - delete this; - } - - std::shared_ptr getData(size_t width, size_t height) override; -}; -} // namespace FormatReader -#endif \ No newline at end of file diff --git a/demos/common/format_reader/register.h b/demos/common/format_reader/register.h deleted file mode 100644 index 34cf1f77f20..00000000000 --- a/demos/common/format_reader/register.h +++ /dev/null @@ -1,56 +0,0 @@ -// Copyright (C) 2018-2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// -/** - * \brief Register for readers - * \file register.h - */ -#pragma once - -#include -#include -#include -#include - -namespace FormatReader { -/** - * \class Registry - * \brief Create reader from fabric - */ -class Registry { -private: - typedef std::function CreatorFunction; - static std::vector _data; -public: - /** - * \brief Create reader - * @param filename - path to input data - * @return Reader for input data or nullptr - */ - static Reader *CreateReader(const char *filename); - - /** - * \brief Registers reader in fabric - * @param f - a creation function - */ - static void RegisterReader(CreatorFunction f); -}; - -/** - * \class Register - * \brief Registers reader in fabric - */ -template -class Register { -public: - /** - * \brief Constructor creates creation function for fabric - * @return Register object - */ - Register() { - Registry::RegisterReader([](const std::string &filename) -> Reader * { - return new To(filename); - }); - } -}; -} // namespace FormatReader \ No newline at end of file diff --git a/demos/common/monitors/CMakeLists.txt b/demos/common/monitors/CMakeLists.txt new file mode 100644 index 00000000000..df429408a13 --- /dev/null +++ b/demos/common/monitors/CMakeLists.txt @@ -0,0 +1,23 @@ +# Copyright (C) 2018-2019 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# + +find_package(OpenCV REQUIRED COMPONENTS core imgproc) + +set(SOURCES presenter.cpp cpu_monitor.cpp memory_monitor.cpp) +set(HEADERS presenter.h cpu_monitor.h memory_monitor.h) +if(WIN32) + list(APPEND SOURCES query_wrapper.cpp) + list(APPEND HEADERS query_wrapper.h) +endif() +# Create named folders for the sources within the .vcproj +# Empty name lists them directly under the .vcproj +source_group("src" FILES ${SOURCES}) +source_group("include" FILES ${HEADERS}) + +add_library(monitors STATIC ${SOURCES} ${HEADERS}) +target_include_directories(monitors PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}") +target_link_libraries(monitors PRIVATE opencv_core opencv_imgproc) +if(WIN32) + target_link_libraries(monitors PRIVATE pdh) +endif() diff --git a/demos/common/monitors/cpu_monitor.cpp b/demos/common/monitors/cpu_monitor.cpp new file mode 100644 index 00000000000..00a841287dd --- /dev/null +++ b/demos/common/monitors/cpu_monitor.cpp @@ -0,0 +1,199 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "cpu_monitor.h" +#include +#ifdef _WIN32 +#include "query_wrapper.h" +#include +#include +#include +#include + +namespace { +const std::size_t nCores = []() { + SYSTEM_INFO sysinfo; + GetSystemInfo(&sysinfo); + return sysinfo.dwNumberOfProcessors; + }(); +} + +class CpuMonitor::PerformanceCounter { +public: + PerformanceCounter() : coreTimeCounters(nCores) { + PDH_STATUS status; + for (std::size_t i = 0; i < nCores; ++i) { + std::wstring fullCounterPath{L"\\Processor(" + std::to_wstring(i) + L")\\% Processor Time"}; + status = PdhAddCounterW(query, fullCounterPath.c_str(), 0, &coreTimeCounters[i]); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhAddCounterW() failed"); + } + status = PdhSetCounterScaleFactor(coreTimeCounters[i], -2); // scale counter to [0, 1] + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhSetCounterScaleFactor() failed"); + } + } + status = PdhCollectQueryData(query); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhCollectQueryData() failed"); + } + } + + std::vector getCpuLoad() { + PDH_STATUS status; + status = PdhCollectQueryData(query); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhCollectQueryData() failed"); + } + + PDH_FMT_COUNTERVALUE displayValue; + std::vector cpuLoad(coreTimeCounters.size()); + for (std::size_t i = 0; i < coreTimeCounters.size(); ++i) { + status = PdhGetFormattedCounterValue(coreTimeCounters[i], PDH_FMT_DOUBLE, NULL, + &displayValue); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhGetFormattedCounterValue() failed"); + } + if (PDH_CSTATUS_VALID_DATA != displayValue.CStatus && PDH_CSTATUS_NEW_DATA != displayValue.CStatus) { + throw std::runtime_error("Error in counter data"); + } + + cpuLoad[i] = displayValue.doubleValue; + } + return cpuLoad; + } + +private: + QueryWrapper query; + std::vector coreTimeCounters; +}; + +#elif __linux__ +#include +#include +#include +#include +#include + +namespace { +const long clockTicks = sysconf(_SC_CLK_TCK); + +const std::size_t nCores = sysconf(_SC_NPROCESSORS_CONF); + +std::vector getIdleCpuStat() { + std::vector idleCpuStat(nCores); + std::ifstream procStat("/proc/stat"); + std::string line; + std::smatch match; + std::regex coreJiffies("^cpu(\\d+)\\s+" + "(\\d+)\\s+" + "(\\d+)\\s+" + "(\\d+)\\s+" + "(\\d+)\\s+" // idle + "(\\d+)"); // iowait + + while (std::getline(procStat, line)) { + if (std::regex_search(line, match, coreJiffies)) { + // it doesn't handle overflow of sum and overflows of /proc/stat values + unsigned long idleInfo = stoul(match[5]) + stoul(match[6]), + coreId = stoul(match[1]); + if (nCores <= coreId) { + throw std::runtime_error("The number of cores has changed"); + } + idleCpuStat[coreId] = idleInfo; + } + } + return idleCpuStat; +} +} + +class CpuMonitor::PerformanceCounter { +public: + PerformanceCounter() : prevIdleCpuStat{getIdleCpuStat()}, prevTimePoint{std::chrono::steady_clock::now()} {} + + std::vector getCpuLoad() { + std::vector idleCpuStat = getIdleCpuStat(); + auto timePoint = std::chrono::steady_clock::now(); + // don't update data too frequently which may result in negative values for cpuLoad. + // It may happen when collectData() is called just after setHistorySize(). + if (timePoint - prevTimePoint > std::chrono::milliseconds{100}) { + std::vector cpuLoad(nCores); + for (std::size_t i = 0; i < idleCpuStat.size(); ++i) { + double idleDiff = idleCpuStat[i] - prevIdleCpuStat[i]; + typedef std::chrono::duration Sec; + cpuLoad[i] = 1.0 + - idleDiff / clockTicks / std::chrono::duration_cast(timePoint - prevTimePoint).count(); + } + return cpuLoad; + } + return {}; + } +private: + std::vector prevIdleCpuStat; + std::chrono::steady_clock::time_point prevTimePoint; +}; + +#else +// not implemented +namespace { +const std::size_t nCores{0}; +} + +class CpuMonitor::PerformanceCounter { +public: + std::vector getCpuLoad() {return {};}; +}; +#endif + +CpuMonitor::CpuMonitor() : + samplesNumber{0}, + historySize{0}, + cpuLoadSum(nCores, 0) {} + +// PerformanceCounter is incomplete in header and destructor can't be defined implicitly +CpuMonitor::~CpuMonitor() = default; + +void CpuMonitor::setHistorySize(std::size_t size) { + if (0 == historySize && 0 != size) { + performanceCounter.reset(new PerformanceCounter); + } else if (0 != historySize && 0 == size) { + performanceCounter.reset(); + } + historySize = size; + std::size_t newSize = std::min(size, cpuLoadHistory.size()); + cpuLoadHistory.erase(cpuLoadHistory.begin(), cpuLoadHistory.end() - newSize); +} + +void CpuMonitor::collectData() { + std::vector cpuLoad = performanceCounter->getCpuLoad(); + + if (!cpuLoad.empty()) { + for (std::size_t i = 0; i < cpuLoad.size(); ++i) { + cpuLoadSum[i] += cpuLoad[i]; + } + ++samplesNumber; + + cpuLoadHistory.push_back(std::move(cpuLoad)); + if (cpuLoadHistory.size() > historySize) { + cpuLoadHistory.pop_front(); + } + } +} + +std::size_t CpuMonitor::getHistorySize() const { + return historySize; +} + +std::deque> CpuMonitor::getLastHistory() const { + return cpuLoadHistory; +} + +std::vector CpuMonitor::getMeanCpuLoad() const { + std::vector meanCpuLoad; + meanCpuLoad.reserve(cpuLoadSum.size()); + for (double coreLoad : cpuLoadSum) { + meanCpuLoad.push_back(coreLoad / samplesNumber); + } + return meanCpuLoad; +} diff --git a/demos/common/monitors/cpu_monitor.h b/demos/common/monitors/cpu_monitor.h new file mode 100644 index 00000000000..38d284538e4 --- /dev/null +++ b/demos/common/monitors/cpu_monitor.h @@ -0,0 +1,28 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include + +class CpuMonitor { +public: + CpuMonitor(); + ~CpuMonitor(); + void setHistorySize(std::size_t size); + std::size_t getHistorySize() const; + void collectData(); + std::deque> getLastHistory() const; + std::vector getMeanCpuLoad() const; + +private: + unsigned samplesNumber; + unsigned historySize; + std::vector cpuLoadSum; + std::deque> cpuLoadHistory; + class PerformanceCounter; + std::unique_ptr performanceCounter; +}; diff --git a/demos/common/monitors/memory_monitor.cpp b/demos/common/monitors/memory_monitor.cpp new file mode 100644 index 00000000000..74019b46d37 --- /dev/null +++ b/demos/common/monitors/memory_monitor.cpp @@ -0,0 +1,208 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "memory_monitor.h" + +struct MemState { + double memTotal, usedMem, usedSwap; +}; + +#ifdef _WIN32 +#include "query_wrapper.h" +#include +#define PSAPI_VERSION 2 +#include +#include +#include +#include + +namespace { +double getMemTotal() { + PERFORMANCE_INFORMATION performanceInformation; + if (!GetPerformanceInfo(&performanceInformation, sizeof(performanceInformation))) { + throw std::runtime_error("GetPerformanceInfo() failed"); + } + return static_cast(performanceInformation.PhysicalTotal * performanceInformation.PageSize) + / (1024 * 1024 * 1024); +} +} + +class MemoryMonitor::PerformanceCounter { +public: + PerformanceCounter() { + PDH_STATUS status = PdhAddCounterW(query, L"\\Paging File(_Total)\\% Usage", 0, &pagingFileUsageCounter); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhAddCounterW() failed"); + } + status = PdhSetCounterScaleFactor(pagingFileUsageCounter, -2); // scale counter to [0, 1] + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhSetCounterScaleFactor() failed"); + } + } + + MemState getMemState() { + PERFORMANCE_INFORMATION performanceInformation; + if (!GetPerformanceInfo(&performanceInformation, sizeof(performanceInformation))) { + throw std::runtime_error("GetPerformanceInfo() failed"); + } + + PDH_STATUS status; + status = PdhCollectQueryData(query); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhCollectQueryData() failed"); + } + PDH_FMT_COUNTERVALUE displayValue; + status = PdhGetFormattedCounterValue(pagingFileUsageCounter, PDH_FMT_DOUBLE, NULL, &displayValue); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhGetFormattedCounterValue() failed"); + } + if (PDH_CSTATUS_VALID_DATA != displayValue.CStatus && PDH_CSTATUS_NEW_DATA != displayValue.CStatus) { + throw std::runtime_error("Error in counter data"); + } + + double pagingFilesSize = static_cast( + (performanceInformation.CommitLimit - performanceInformation.PhysicalTotal) + * performanceInformation.PageSize) / (1024 * 1024 * 1024); + return {static_cast(performanceInformation.PhysicalTotal * performanceInformation.PageSize) + / (1024 * 1024 * 1024), + static_cast( + (performanceInformation.PhysicalTotal - performanceInformation.PhysicalAvailable) + * performanceInformation.PageSize) / (1024 * 1024 * 1024), + pagingFilesSize * displayValue.doubleValue}; + } +private: + QueryWrapper query; + PDH_HCOUNTER pagingFileUsageCounter; +}; + +#elif __linux__ +#include +#include +#include +#include + +namespace { +std::pair, std::pair> getAvailableMemSwapTotalMemSwap() { + double memAvailable = 0, swapFree = 0, memTotal = 0, swapTotal = 0; + std::regex memRegex("^(.+):\\s+(\\d+) kB$"); + std::string line; + std::smatch match; + std::ifstream meminfo("/proc/meminfo"); + while (std::getline(meminfo, line)) { + if (std::regex_match(line, match, memRegex)) { + if ("MemAvailable" == match[1]) { + memAvailable = stod(match[2]) / (1024 * 1024); + } else if ("SwapFree" == match[1]) { + swapFree = stod(match[2]) / (1024 * 1024); + } else if ("MemTotal" == match[1]) { + memTotal = stod(match[2]) / (1024 * 1024); + } else if ("SwapTotal" == match[1]) { + swapTotal = stod(match[2]) / (1024 * 1024); + } + } + } + if (0 == memTotal) { + throw std::runtime_error("Can't get MemTotal"); + } + return {{memAvailable, swapFree}, {memTotal, swapTotal}}; +} + +double getMemTotal() { + return getAvailableMemSwapTotalMemSwap().second.first; +} +} + +class MemoryMonitor::PerformanceCounter { +public: + MemState getMemState() { + std::pair, std::pair> availableMemSwapTotalMemSwap + = getAvailableMemSwapTotalMemSwap(); + double memTotal = availableMemSwapTotalMemSwap.second.first; + double swapTotal = availableMemSwapTotalMemSwap.second.second; + return {memTotal, memTotal - availableMemSwapTotalMemSwap.first.first, swapTotal - availableMemSwapTotalMemSwap.first.second}; + } +}; + +#else +// not implemented +namespace { +double getMemTotal() {return 0.0;} +} + +class MemoryMonitor::PerformanceCounter { +public: + MemState getMemState() {return {0.0, 0.0, 0.0};} +}; +#endif + +MemoryMonitor::MemoryMonitor() : + samplesNumber{0}, + historySize{0}, + memSum{0.0}, + swapSum{0.0}, + maxMem{0.0}, + maxSwap{0.0}, + memTotal{::getMemTotal()}, + maxMemTotal{memTotal} {} + +// PerformanceCounter is incomplete in header and destructor can't be defined implicitly +MemoryMonitor::~MemoryMonitor() = default; + +void MemoryMonitor::setHistorySize(std::size_t size) { + if (0 == historySize && 0 != size) { + performanceCounter.reset(new MemoryMonitor::PerformanceCounter); + } else if (0 != historySize && 0 == size) { + performanceCounter.reset(); + } + historySize = size; + std::size_t newSize = std::min(size, memSwapUsageHistory.size()); + memSwapUsageHistory.erase(memSwapUsageHistory.begin(), memSwapUsageHistory.end() - newSize); +} + +void MemoryMonitor::collectData() { + MemState memState = performanceCounter->getMemState(); + maxMemTotal = std::max(maxMemTotal, memState.memTotal); + memSum += memState.usedMem; + swapSum += memState.usedSwap; + ++samplesNumber; + maxMem = std::max(maxMem, memState.usedMem); + maxSwap = std::max(maxSwap, memState.usedSwap); + + memSwapUsageHistory.emplace_back(memState.usedMem, memState.usedSwap); + if (memSwapUsageHistory.size() > historySize) { + memSwapUsageHistory.pop_front(); + } +} + +std::size_t MemoryMonitor::getHistorySize() const { + return historySize; +} + +std::deque> MemoryMonitor::getLastHistory() const { + return memSwapUsageHistory; +} + +double MemoryMonitor::getMeanMem() const { + return memSum / samplesNumber; +} + +double MemoryMonitor::getMeanSwap() const { + return swapSum / samplesNumber; +} + +double MemoryMonitor::getMaxMem() const { + return maxMem; +} + +double MemoryMonitor::getMaxSwap() const { + return maxSwap; +} + +double MemoryMonitor::getMemTotal() const { + return memTotal; +} + +double MemoryMonitor::getMaxMemTotal() const { + return maxMemTotal; +} diff --git a/demos/common/monitors/memory_monitor.h b/demos/common/monitors/memory_monitor.h new file mode 100644 index 00000000000..9eda10f7559 --- /dev/null +++ b/demos/common/monitors/memory_monitor.h @@ -0,0 +1,34 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include + +class MemoryMonitor { +public: + MemoryMonitor(); + ~MemoryMonitor(); + void setHistorySize(std::size_t size); + std::size_t getHistorySize() const; + void collectData(); + std::deque> getLastHistory() const; + double getMeanMem() const; // in GiB + double getMeanSwap() const; + double getMaxMem() const; + double getMaxSwap() const; + double getMemTotal() const; + double getMaxMemTotal() const; // a system may have hotpluggable memory +private: + unsigned samplesNumber; + std::size_t historySize; + double memSum, swapSum; + double maxMem, maxSwap; + double memTotal; + double maxMemTotal; + std::deque> memSwapUsageHistory; + class PerformanceCounter; + std::unique_ptr performanceCounter; +}; diff --git a/demos/common/monitors/presenter.cpp b/demos/common/monitors/presenter.cpp new file mode 100644 index 00000000000..0cadc9d5279 --- /dev/null +++ b/demos/common/monitors/presenter.cpp @@ -0,0 +1,301 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include +#include +#include + +#include "presenter.h" + +namespace { +const std::map keyToMonitorType{ + {'C', MonitorType::CpuAverage}, + {'D', MonitorType::DistributionCpu}, + {'M', MonitorType::Memory}}; + +std::set strKeysToMonitorSet(const std::string& keys) { + std::set enabledMonitors; + for (unsigned char key: keys) { + auto iter = keyToMonitorType.find(std::toupper(key)); + if (keyToMonitorType.end() == iter) { + throw std::runtime_error("Unknown monitor type"); + } else { + enabledMonitors.insert(iter->second); + } + } + return enabledMonitors; +} +} + +Presenter::Presenter(std::set enabledMonitors, + int yPos, + cv::Size graphSize, + std::size_t historySize) : + yPos{yPos}, + graphSize{graphSize}, + graphPadding{std::max(1, static_cast(graphSize.width * 0.05))}, + historySize{historySize}, + distributionCpuEnabled{false}, + strStream{std::ios_base::app} { + for (MonitorType monitor : enabledMonitors) { + addRemoveMonitor(monitor); + } +} + +Presenter::Presenter(const std::string& keys, int yPos, cv::Size graphSize, std::size_t historySize) : + Presenter{strKeysToMonitorSet(keys), yPos, graphSize, historySize} {} + +void Presenter::addRemoveMonitor(MonitorType monitor) { + unsigned updatedHistorySize = 1; + if (historySize > 1) { + int sampleStep = std::max(1, static_cast(graphSize.width / (historySize - 1))); + // +1 to plot graphSize.width/sampleStep segments + // add round up to and an extra element if don't reach graph edge + updatedHistorySize = (graphSize.width + sampleStep - 1) / sampleStep + 1; + } + switch(monitor) { + case MonitorType::CpuAverage: { + if (cpuMonitor.getHistorySize() > 1 && distributionCpuEnabled) { + cpuMonitor.setHistorySize(1); + } else if (cpuMonitor.getHistorySize() > 1 && !distributionCpuEnabled) { + cpuMonitor.setHistorySize(0); + } else { // cpuMonitor.getHistorySize() <= 1 + cpuMonitor.setHistorySize(updatedHistorySize); + } + break; + } + case MonitorType::DistributionCpu: { + if (distributionCpuEnabled) { + distributionCpuEnabled = false; + if (1 == cpuMonitor.getHistorySize()) { // cpuMonitor was used only for DistributionCpu => disable it + cpuMonitor.setHistorySize(0); + } + } else { + distributionCpuEnabled = true; + cpuMonitor.setHistorySize(std::max(std::size_t{1}, cpuMonitor.getHistorySize())); + } + break; + } + case MonitorType::Memory: { + if (memoryMonitor.getHistorySize() > 1) { + memoryMonitor.setHistorySize(0); + } else { + memoryMonitor.setHistorySize(updatedHistorySize); + } + break; + } + } +} + +void Presenter::handleKey(int key) { + key = std::toupper(key); + if ('H' == key) { + if (0 == cpuMonitor.getHistorySize() && memoryMonitor.getHistorySize() <= 1) { + addRemoveMonitor(MonitorType::CpuAverage); + addRemoveMonitor(MonitorType::DistributionCpu); + addRemoveMonitor(MonitorType::Memory); + } else { + cpuMonitor.setHistorySize(0); + distributionCpuEnabled = false; + memoryMonitor.setHistorySize(0); + } + } else { + auto iter = keyToMonitorType.find(key); + if (keyToMonitorType.end() != iter) { + addRemoveMonitor(iter->second); + } + } +} + +void Presenter::drawGraphs(cv::Mat& frame) { + const std::chrono::steady_clock::time_point curTimeStamp = std::chrono::steady_clock::now(); + if (curTimeStamp - prevTimeStamp >= std::chrono::milliseconds{1000}) { + prevTimeStamp = curTimeStamp; + if (0 != cpuMonitor.getHistorySize()) { + cpuMonitor.collectData(); + } + if (memoryMonitor.getHistorySize() > 1) { + memoryMonitor.collectData(); + } + } + + int numberOfEnabledMonitors = (cpuMonitor.getHistorySize() > 1) + distributionCpuEnabled + + (memoryMonitor.getHistorySize() > 1); + int panelWidth = graphSize.width * numberOfEnabledMonitors + + std::max(0, numberOfEnabledMonitors - 1) * graphPadding; + while (panelWidth > frame.cols) { + panelWidth = std::max(0, panelWidth - graphSize.width - graphPadding); + --numberOfEnabledMonitors; // can't draw all monitors + } + int graphPos = std::max(0, (frame.cols - 1 - panelWidth) / 2); + int textGraphSplittingLine = graphSize.height / 5; + int graphRectHeight = graphSize.height - textGraphSplittingLine; + int sampleStep = 1; + unsigned possibleHistorySize = 1; + if (historySize > 1) { + sampleStep = std::max(1, static_cast(graphSize.width / (historySize - 1))); + possibleHistorySize = (graphSize.width + sampleStep - 1) / sampleStep + 1; + } + + if (cpuMonitor.getHistorySize() > 1 && possibleHistorySize > 1 && --numberOfEnabledMonitors >= 0) { + std::deque> lastHistory = cpuMonitor.getLastHistory(); + cv::Mat graph = frame(cv::Rect{cv::Point{graphPos, yPos}, graphSize} & cv::Rect(0, 0, frame.cols, frame.rows)); + graph = graph / 2 + cv::Scalar{127, 127, 127}; + + int lineXPos = graph.cols - 1; + std::vector averageLoad(lastHistory.size()); + + for (int i = lastHistory.size() - 1; i >= 0; --i) { + double mean = std::accumulate(lastHistory[i].begin(), lastHistory[i].end(), 0.0) / lastHistory[i].size(); + averageLoad[i] = {lineXPos, graphSize.height - static_cast(mean * graphRectHeight)}; + lineXPos -= sampleStep; + } + + cv::polylines(graph, averageLoad, false, {255, 0, 0}, 2); + cv::rectangle(frame, cv::Rect{ + cv::Point{graphPos, yPos + textGraphSplittingLine}, + cv::Size{graphSize.width, graphSize.height - textGraphSplittingLine} + }, {0, 0, 0}); + strStream.str("CPU"); + if (!lastHistory.empty()) { + strStream << ": " << std::fixed << std::setprecision(1) + << std::accumulate(lastHistory.back().begin(), lastHistory.back().end(), 0.0) + / lastHistory.back().size() * 100 << '%'; + } + int baseline; + int textWidth = cv::getTextSize(strStream.str(), + cv::FONT_HERSHEY_SIMPLEX, + textGraphSplittingLine * 0.04, + 1, + &baseline).width; + cv::putText(graph, + strStream.str(), + cv::Point{(graphSize.width - textWidth) / 2, textGraphSplittingLine - 1}, + cv::FONT_HERSHEY_SIMPLEX, + textGraphSplittingLine * 0.04, + {70, 0, 0}, + 1); + graphPos += graphSize.width + graphPadding; + } + + if (distributionCpuEnabled && --numberOfEnabledMonitors >= 0) { + std::deque> lastHistory = cpuMonitor.getLastHistory(); + cv::Mat graph = frame(cv::Rect{cv::Point{graphPos, yPos}, graphSize} & cv::Rect(0, 0, frame.cols, frame.rows)); + graph = graph / 2 + cv::Scalar{127, 127, 127}; + + if (!lastHistory.empty()) { + int rectXPos = 0; + int step = (graph.cols + lastHistory.back().size() - 1) / lastHistory.back().size(); // round up + double sum = 0; + for (double coreLoad : lastHistory.back()) { + sum += coreLoad; + int height = static_cast(graphRectHeight * coreLoad); + cv::Rect pillar{cv::Point{rectXPos, graph.rows - height}, cv::Size{step, height}}; + cv::rectangle(graph, pillar, {255, 0, 0}, cv::FILLED); + cv::rectangle(graph, pillar, {0, 0, 0}); + rectXPos += step; + } + sum /= lastHistory.back().size(); + int yLine = graph.rows - static_cast(graphRectHeight * sum); + cv::line(graph, cv::Point{0, yLine}, cv::Point{graph.cols, yLine}, {0, 255, 0}, 2); + } + cv::Rect border{cv::Point{graphPos, yPos + textGraphSplittingLine}, + cv::Size{graphSize.width, graphSize.height - textGraphSplittingLine}}; + cv::rectangle(frame, border, {0, 0, 0}); + strStream.str("Core load"); + if (!lastHistory.empty()) { + strStream << ": " << std::fixed << std::setprecision(1) + << std::accumulate(lastHistory.back().begin(), lastHistory.back().end(), 0.0) + / lastHistory.back().size() * 100 << '%'; + } + int baseline; + int textWidth = cv::getTextSize(strStream.str(), + cv::FONT_HERSHEY_SIMPLEX, + textGraphSplittingLine * 0.04, + 1, + &baseline).width; + cv::putText(graph, + strStream.str(), + cv::Point{(graphSize.width - textWidth) / 2, textGraphSplittingLine - 1}, + cv::FONT_HERSHEY_SIMPLEX, + textGraphSplittingLine * 0.04, + {0, 70, 0}); + graphPos += graphSize.width + graphPadding; + } + + if (memoryMonitor.getHistorySize() > 1 && possibleHistorySize > 1 && --numberOfEnabledMonitors >= 0) { + std::deque> lastHistory = memoryMonitor.getLastHistory(); + cv::Mat graph = frame(cv::Rect{cv::Point{graphPos, yPos}, graphSize} & cv::Rect(0, 0, frame.cols, frame.rows)); + graph = graph / 2 + cv::Scalar{127, 127, 127}; + int histxPos = graph.cols - 1; + double range = std::min(memoryMonitor.getMaxMemTotal() + memoryMonitor.getMaxSwap(), + (memoryMonitor.getMaxMem() + memoryMonitor.getMaxSwap()) * 1.2); + if (lastHistory.size() > 1) { + for (auto memUsageIt = lastHistory.rbegin(); memUsageIt != lastHistory.rend() - 1; ++memUsageIt) { + constexpr double SWAP_THRESHOLD = 10.0 / 1024; // 10 MiB + cv::Vec3b color = + (memoryMonitor.getMemTotal() * 0.95 > memUsageIt->first) || (memUsageIt->second < SWAP_THRESHOLD) ? + cv::Vec3b{0, 255, 255} : + cv::Vec3b{0, 0, 255}; + cv::Point right{histxPos, + graph.rows - static_cast(graphRectHeight * (memUsageIt->first + memUsageIt->second) / range)}; + cv::Point left{histxPos - sampleStep, + graph.rows - static_cast( + graphRectHeight * ((memUsageIt + 1)->first + (memUsageIt + 1)->second) / range)}; + cv::line(graph, right, left, color, 2); + histxPos -= sampleStep; + } + } + + cv::Rect border{cv::Point{graphPos, yPos + textGraphSplittingLine}, + cv::Size{graphSize.width, graphSize.height - textGraphSplittingLine}}; + cv::rectangle(frame, {border}, {0, 0, 0}); + if (lastHistory.empty()) { + strStream.str("Memory"); + } else { + strStream.str(""); + strStream << std::fixed << std::setprecision(1) << lastHistory.back().first << " + " + << lastHistory.back().second << " GiB"; + } + int baseline; + int textWidth = cv::getTextSize(strStream.str(), + cv::FONT_HERSHEY_SIMPLEX, + textGraphSplittingLine * 0.04, + 1, + &baseline).width; + cv::putText(graph, + strStream.str(), + cv::Point{(graphSize.width - textWidth) / 2, textGraphSplittingLine - 1}, + cv::FONT_HERSHEY_SIMPLEX, + textGraphSplittingLine * 0.04, + {0, 35, 35}); + } +} + +std::string Presenter::reportMeans() const { + std::ostringstream collectedDataStream; + collectedDataStream << std::fixed << std::setprecision(1); + if (cpuMonitor.getHistorySize() > 1) { + collectedDataStream << "Mean core utilization: "; + for (double mean : cpuMonitor.getMeanCpuLoad()) { + collectedDataStream << mean * 100 << "% "; + } + collectedDataStream << '\n'; + } + if (distributionCpuEnabled) { + std::vector meanCpuLoad = cpuMonitor.getMeanCpuLoad(); + double mean = std::accumulate(meanCpuLoad.begin(), meanCpuLoad.end(), 0.0) / meanCpuLoad.size(); + collectedDataStream << "Mean CPU utilization: " << mean * 100 << "%\n"; + } + if (memoryMonitor.getHistorySize() > 1) { + collectedDataStream << "Memory mean usage: " << memoryMonitor.getMeanMem() << " GiB\n"; + collectedDataStream << "Mean swap usage: " << memoryMonitor.getMeanSwap() << " GiB\n"; + } + std::string collectedData = collectedDataStream.str(); + // drop last \n because usually it is not expeted that printing an object starts a new line + if (!collectedData.empty()) { + return collectedData.substr(0, collectedData.size() - 1); + } + return collectedData; +} diff --git a/demos/common/monitors/presenter.h b/demos/common/monitors/presenter.h new file mode 100644 index 00000000000..bb1c9566f98 --- /dev/null +++ b/demos/common/monitors/presenter.h @@ -0,0 +1,43 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include + +#include + +#include "cpu_monitor.h" +#include "memory_monitor.h" + +enum class MonitorType{CpuAverage, DistributionCpu, Memory}; + +class Presenter { +public: + explicit Presenter(std::set enabledMonitors = {}, + int yPos = 20, + cv::Size graphSize = {150, 60}, + std::size_t historySize = 20); + explicit Presenter(const std::string& keys, + int yPos = 20, + cv::Size graphSize = {150, 60}, + std::size_t historySize = 20); + void addRemoveMonitor(MonitorType monitor); + void handleKey(int key); // handles c, d, m, h keys + void drawGraphs(cv::Mat& frame); + std::string reportMeans() const; + + const int yPos; + const cv::Size graphSize; + const int graphPadding; +private: + std::chrono::steady_clock::time_point prevTimeStamp; + std::size_t historySize; + CpuMonitor cpuMonitor; + bool distributionCpuEnabled; + MemoryMonitor memoryMonitor; + std::ostringstream strStream; +}; diff --git a/demos/common/monitors/query_wrapper.cpp b/demos/common/monitors/query_wrapper.cpp new file mode 100644 index 00000000000..184a0e65ac4 --- /dev/null +++ b/demos/common/monitors/query_wrapper.cpp @@ -0,0 +1,22 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "query_wrapper.h" + +#include +#include + +QueryWrapper::QueryWrapper() { + PDH_STATUS status = PdhOpenQuery(NULL, NULL, &query); + if (ERROR_SUCCESS != status) { + throw std::system_error(status, std::system_category(), "PdhOpenQuery() failed"); + } +} +QueryWrapper::~QueryWrapper() { + PdhCloseQuery(query); +} + +QueryWrapper::operator PDH_HQUERY() const { + return query; +} diff --git a/demos/common/monitors/query_wrapper.h b/demos/common/monitors/query_wrapper.h new file mode 100644 index 00000000000..ea0e5599e5b --- /dev/null +++ b/demos/common/monitors/query_wrapper.h @@ -0,0 +1,17 @@ +// Copyright (C) 2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +class QueryWrapper { +public: + QueryWrapper(); + ~QueryWrapper(); + QueryWrapper(const QueryWrapper&) = delete; + QueryWrapper& operator=(const QueryWrapper&) = delete; + operator PDH_HQUERY() const; +private: + PDH_HQUERY query; +}; diff --git a/demos/common/samples/args_helper.hpp b/demos/common/samples/args_helper.hpp index fc193aeeab9..0fe9982533d 100644 --- a/demos/common/samples/args_helper.hpp +++ b/demos/common/samples/args_helper.hpp @@ -34,7 +34,7 @@ inline void readInputFilesArguments(std::vector &files, const std::string& arg) { struct stat sb; if (stat(arg.c_str(), &sb) != 0) { - if (arg.find("rtsp:") != 0) { + if (arg.compare(0, 5, "rtsp:") != 0) { slog::warn << "File " << arg << " cannot be opened!" << slog::endl; return; } @@ -103,15 +103,18 @@ inline std::vector split(const std::string &s, char delim) { } inline std::vector parseDevices(const std::string& device_string) { - std::string comma_separated_devices = device_string; - const std::string::size_type colon_position = comma_separated_devices.find(":"); + const std::string::size_type colon_position = device_string.find(":"); if (colon_position != std::string::npos) { - comma_separated_devices = comma_separated_devices.substr(colon_position + 1); + std::string device_type = device_string.substr(0, colon_position); + if (device_type == "HETERO" || device_type == "MULTI") { + std::string comma_separated_devices = device_string.substr(colon_position + 1); + std::vector devices = split(comma_separated_devices, ','); + for (auto& device : devices) + device = device.substr(0, device.find("(")); + return devices; + } } - auto devices = split(comma_separated_devices, ','); - for (auto& device : devices) - device = device.substr(0, device.find("(")); - return devices; + return {device_string}; } inline std::map parseValuePerDevice(const std::set& devices, diff --git a/demos/common/samples/classification_results.h b/demos/common/samples/classification_results.h deleted file mode 100644 index 3cf0a2b06be..00000000000 --- a/demos/common/samples/classification_results.h +++ /dev/null @@ -1,92 +0,0 @@ -// Copyright (C) 2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -/** - * @brief a header file with ouput classification results - * @file classification_results.hpp - */ -#include -#include -#include -#include - -#include - -/** - * @class ClassificationResult - * @brief A ClassificationResult creates an output table with results - */ -class ClassificationResult { -private: - const std::string _classidStr = "classid"; - const std::string _probabilityStr = "probability"; - const std::string _labelStr = "label"; - size_t _nTop; - InferenceEngine::Blob::Ptr _outBlob; - const std::vector _labels; - const std::vector _imageNames; - const size_t _batchSize; - - void printHeader() { - std::cout << _classidStr << " " << _probabilityStr; - if (!_labels.empty()) - std::cout << " " << _labelStr; - std::string classidColumn(_classidStr.length(), '-'); - std::string probabilityColumn(_probabilityStr.length(), '-'); - std::string labelColumn(_labelStr.length(), '-'); - std::cout << std::endl << classidColumn << " " << probabilityColumn; - if (!_labels.empty()) - std::cout << " " << labelColumn; - std::cout << std::endl; - } - -public: - explicit ClassificationResult(InferenceEngine::Blob::Ptr output_blob, - std::vector image_names = {}, - size_t batch_size = 1, - size_t num_of_top = 10, - std::vector labels = {}) : - _nTop(num_of_top), - _outBlob(std::move(output_blob)), - _labels(std::move(labels)), - _imageNames(std::move(image_names)), - _batchSize(batch_size) { - if (_imageNames.size() != _batchSize) { - throw std::logic_error("Batch size should be equal to the number of images."); - } - } - - /** - * @brief prints formatted classification results - */ - void print() { - /** This vector stores id's of top N results **/ - std::vector results; - TopResults(_nTop, *_outBlob, results); - - /** Print the result iterating over each batch **/ - std::cout << std::endl << "Top " << _nTop << " results:" << std::endl << std::endl; - for (unsigned int image_id = 0; image_id < _batchSize; ++image_id) { - std::cout << "Image " << _imageNames[image_id] << std::endl << std::endl; - printHeader(); - - for (size_t id = image_id * _nTop, cnt = 0; id < (image_id + 1) * _nTop; ++cnt, ++id) { - std::cout.precision(7); - /** Getting probability for resulting class **/ - const auto result = _outBlob->buffer(). - as::value_type*>() - [results[id] + image_id * (_outBlob->size() / _batchSize)]; - - std::cout << std::setw(static_cast(_classidStr.length())) << std::left << results[id] << " "; - std::cout << std::left << std::setw(static_cast(_probabilityStr.length())) << std::fixed << result; - - if (!_labels.empty()) { - std::cout << " " + _labels[results[id]]; - } - std::cout << std::endl; - } - std::cout << std::endl; - } - } -}; diff --git a/demos/common/samples/common.hpp b/demos/common/samples/common.hpp index 1b48f329f9c..7a8542a5975 100644 --- a/demos/common/samples/common.hpp +++ b/demos/common/samples/common.hpp @@ -48,15 +48,9 @@ class ConsoleErrorListener : public InferenceEngine::IErrorListener { } }; -/** - * @brief Trims from both ends (in place) - * @param s - string to trim - * @return trimmed string - */ -inline std::string &trim(std::string &s) { - s.erase(s.begin(), std::find_if(s.begin(), s.end(), std::not1(std::ptr_fun(std::isspace)))); - s.erase(std::find_if(s.rbegin(), s.rend(), std::not1(std::ptr_fun(std::isspace))).base(), s.end()); - return s; +template +constexpr std::size_t arraySize(const T (&)[N]) noexcept { + return N; } /** @@ -70,33 +64,6 @@ static UNUSED std::string fileNameNoExt(const std::string &filepath) { return filepath.substr(0, pos); } -/** -* @brief Get extension from filename -* @param filename - name of the file which extension should be extracted -* @return string with extracted file extension -*/ -inline std::string fileExt(const std::string& filename) { - auto pos = filename.rfind('.'); - if (pos == std::string::npos) return ""; - return filename.substr(pos + 1); -} - -static UNUSED std::ostream &operator<<(std::ostream &os, const InferenceEngine::Version *version) { - os << "\n\tAPI version ............ "; - if (nullptr == version) { - os << "UNKNOWN"; - } else { - os << version->apiVersion.major << "." << version->apiVersion.minor; - if (nullptr != version->buildNumber) { - os << "\n\t" << "Build .................. " << version->buildNumber; - } - if (nullptr != version->description) { - os << "\n\t" << "Description ....... " << version->description; - } - } - return os; -} - inline std::ostream &operator<<(std::ostream &os, const InferenceEngine::Version &version) { os << "\t" << version.description << " version ......... "; os << version.apiVersion.major << "." << version.apiVersion.minor; @@ -116,55 +83,6 @@ inline std::ostream &operator<<(std::ostream &os, const std::map> blobToImageOutputArray(InferenceEngine::TBlob::Ptr output, - size_t *pWidth, size_t *pHeight, - size_t *pChannels) { - std::vector> outArray; - size_t W = 0, C = 0, H = 0; - - auto outputDims = output->getTensorDesc().getDims(); - if (outputDims.size() == 3) { - C = outputDims.at(0); - H = outputDims.at(1); - W = outputDims.at(2); - } else if (outputDims.size() == 4) { - C = outputDims.at(1); - H = outputDims.at(2); - W = outputDims.at(3); - } else if (outputDims.size() == 5) { - C = outputDims.at(1); - H = outputDims.at(3); - W = outputDims.at(4); - } else { - THROW_IE_EXCEPTION << "Output blob has unsupported layout " << output->getTensorDesc().getLayout(); - } - - // Get classes - const float *outData = output->data(); - for (unsigned h = 0; h < H; h++) { - std::vector row; - for (unsigned w = 0; w < W; w++) { - float max_value = outData[h * W + w]; - size_t index = 0; - for (size_t c = 1; c < C; c++) { - size_t dataIndex = c * H * W + h * W + w; - if (outData[dataIndex] > max_value) { - index = c; - max_value = outData[dataIndex]; - } - } - row.push_back(index); - } - outArray.push_back(row); - } - - if (pWidth != nullptr) *pWidth = W; - if (pHeight != nullptr) *pHeight = H; - if (pChannels != nullptr) *pChannels = C; - - return outArray; -} - /** * @class Color * @brief A Color class stores channels of a given color @@ -186,385 +104,43 @@ class Color { unsigned char g, unsigned char b) : _r(r), _g(g), _b(b) {} - inline unsigned char red() { + inline unsigned char red() const { return _r; } - inline unsigned char blue() { + inline unsigned char blue() const { return _b; } - inline unsigned char green() { + inline unsigned char green() const { return _g; } }; -// TODO : keep only one version of writeOutputBMP - -/** - * @brief Writes output data to image - * @param name - image name - * @param data - output data - * @param classesNum - the number of classes - * @return false if error else true - */ -static UNUSED void writeOutputBmp(std::vector> data, size_t classesNum, std::ostream &outFile) { - unsigned int seed = (unsigned int) time(NULL); - // Known colors for training classes from Cityscape dataset - static std::vector colors = { - {128, 64, 128}, - {232, 35, 244}, - {70, 70, 70}, - {156, 102, 102}, - {153, 153, 190}, - {153, 153, 153}, - {30, 170, 250}, - {0, 220, 220}, - {35, 142, 107}, - {152, 251, 152}, - {180, 130, 70}, - {60, 20, 220}, - {0, 0, 255}, - {142, 0, 0}, - {70, 0, 0}, - {100, 60, 0}, - {90, 0, 0}, - {230, 0, 0}, - {32, 11, 119}, - {0, 74, 111}, - {81, 0, 81} - }; - - while (classesNum > colors.size()) { - static std::mt19937 rng(seed); - std::uniform_int_distribution dist(0, 255); - Color color(dist(rng), dist(rng), dist(rng)); - colors.push_back(color); - } - - unsigned char file[14] = { - 'B', 'M', // magic - 0, 0, 0, 0, // size in bytes - 0, 0, // app data - 0, 0, // app data - 40 + 14, 0, 0, 0 // start of data offset - }; - unsigned char info[40] = { - 40, 0, 0, 0, // info hd size - 0, 0, 0, 0, // width - 0, 0, 0, 0, // height - 1, 0, // number color planes - 24, 0, // bits per pixel - 0, 0, 0, 0, // compression is none - 0, 0, 0, 0, // image bits size - 0x13, 0x0B, 0, 0, // horz resolution in pixel / m - 0x13, 0x0B, 0, 0, // vert resolution (0x03C3 = 96 dpi, 0x0B13 = 72 dpi) - 0, 0, 0, 0, // #colors in palette - 0, 0, 0, 0, // #important colors - }; - - auto height = data.size(); - auto width = data.at(0).size(); - - if (height > (size_t) std::numeric_limits::max || width > (size_t) std::numeric_limits::max) { - THROW_IE_EXCEPTION << "File size is too big: " << height << " X " << width; - } - - int padSize = static_cast(4 - (width * 3) % 4) % 4; - int sizeData = static_cast(width * height * 3 + height * padSize); - int sizeAll = sizeData + sizeof(file) + sizeof(info); - - file[2] = (unsigned char) (sizeAll); - file[3] = (unsigned char) (sizeAll >> 8); - file[4] = (unsigned char) (sizeAll >> 16); - file[5] = (unsigned char) (sizeAll >> 24); - - info[4] = (unsigned char) (width); - info[5] = (unsigned char) (width >> 8); - info[6] = (unsigned char) (width >> 16); - info[7] = (unsigned char) (width >> 24); - - int32_t negativeHeight = -(int32_t) height; - info[8] = (unsigned char) (negativeHeight); - info[9] = (unsigned char) (negativeHeight >> 8); - info[10] = (unsigned char) (negativeHeight >> 16); - info[11] = (unsigned char) (negativeHeight >> 24); - - info[20] = (unsigned char) (sizeData); - info[21] = (unsigned char) (sizeData >> 8); - info[22] = (unsigned char) (sizeData >> 16); - info[23] = (unsigned char) (sizeData >> 24); - - outFile.write(reinterpret_cast(file), sizeof(file)); - outFile.write(reinterpret_cast(info), sizeof(info)); - - unsigned char pad[3] = {0, 0, 0}; - - for (size_t y = 0; y < height; y++) { - for (size_t x = 0; x < width; x++) { - unsigned char pixel[3]; - size_t index = data.at(y).at(x); - pixel[0] = colors.at(index).red(); - pixel[1] = colors.at(index).green(); - pixel[2] = colors.at(index).blue(); - outFile.write(reinterpret_cast(pixel), 3); - } - outFile.write(reinterpret_cast(pad), padSize); - } -} - -/** -* @brief Writes output data to BMP image -* @param name - image name -* @param data - output data -* @param height - height of the target image -* @param width - width of the target image -* @return false if error else true -*/ -static UNUSED bool writeOutputBmp(std::string name, unsigned char *data, size_t height, size_t width) { - std::ofstream outFile; - outFile.open(name, std::ofstream::binary); - if (!outFile.is_open()) { - return false; - } - - unsigned char file[14] = { - 'B', 'M', // magic - 0, 0, 0, 0, // size in bytes - 0, 0, // app data - 0, 0, // app data - 40 + 14, 0, 0, 0 // start of data offset - }; - unsigned char info[40] = { - 40, 0, 0, 0, // info hd size - 0, 0, 0, 0, // width - 0, 0, 0, 0, // height - 1, 0, // number color planes - 24, 0, // bits per pixel - 0, 0, 0, 0, // compression is none - 0, 0, 0, 0, // image bits size - 0x13, 0x0B, 0, 0, // horz resolution in pixel / m - 0x13, 0x0B, 0, 0, // vert resolution (0x03C3 = 96 dpi, 0x0B13 = 72 dpi) - 0, 0, 0, 0, // #colors in palette - 0, 0, 0, 0, // #important colors - }; - - if (height > (size_t)std::numeric_limits::max || width > (size_t)std::numeric_limits::max) { - THROW_IE_EXCEPTION << "File size is too big: " << height << " X " << width; - } - - int padSize = static_cast(4 - (width * 3) % 4) % 4; - int sizeData = static_cast(width * height * 3 + height * padSize); - int sizeAll = sizeData + sizeof(file) + sizeof(info); - - file[2] = (unsigned char)(sizeAll); - file[3] = (unsigned char)(sizeAll >> 8); - file[4] = (unsigned char)(sizeAll >> 16); - file[5] = (unsigned char)(sizeAll >> 24); - - info[4] = (unsigned char)(width); - info[5] = (unsigned char)(width >> 8); - info[6] = (unsigned char)(width >> 16); - info[7] = (unsigned char)(width >> 24); - - int32_t negativeHeight = -(int32_t)height; - info[8] = (unsigned char)(negativeHeight); - info[9] = (unsigned char)(negativeHeight >> 8); - info[10] = (unsigned char)(negativeHeight >> 16); - info[11] = (unsigned char)(negativeHeight >> 24); - - info[20] = (unsigned char)(sizeData); - info[21] = (unsigned char)(sizeData >> 8); - info[22] = (unsigned char)(sizeData >> 16); - info[23] = (unsigned char)(sizeData >> 24); - - outFile.write(reinterpret_cast(file), sizeof(file)); - outFile.write(reinterpret_cast(info), sizeof(info)); - - unsigned char pad[3] = { 0, 0, 0 }; - - for (size_t y = 0; y < height; y++) { - for (size_t x = 0; x < width; x++) { - unsigned char pixel[3]; - pixel[0] = data[y * width * 3 + x * 3]; - pixel[1] = data[y * width * 3 + x * 3 + 1]; - pixel[2] = data[y * width * 3 + x * 3 + 2]; - - outFile.write(reinterpret_cast(pixel), 3); - } - outFile.write(reinterpret_cast(pad), padSize); - } - return true; -} - - -/** -* @brief Adds colored rectangles to the image -* @param data - data where rectangles are put -* @param height - height of the rectangle -* @param width - width of the rectangle -* @param rectangles - vector points for the rectangle, should be 4x compared to num classes -* @param classes - vector of classes -* @param thickness - thickness of a line (in pixels) to be used for bounding boxes -*/ -static UNUSED void addRectangles(unsigned char *data, size_t height, size_t width, std::vector rectangles, std::vector classes, int thickness = 1) { - std::vector colors = { // colors to be used for bounding boxes - { 128, 64, 128 }, - { 232, 35, 244 }, - { 70, 70, 70 }, - { 156, 102, 102 }, - { 153, 153, 190 }, - { 153, 153, 153 }, - { 30, 170, 250 }, - { 0, 220, 220 }, - { 35, 142, 107 }, - { 152, 251, 152 }, - { 180, 130, 70 }, - { 60, 20, 220 }, - { 0, 0, 255 }, - { 142, 0, 0 }, - { 70, 0, 0 }, - { 100, 60, 0 }, - { 90, 0, 0 }, - { 230, 0, 0 }, - { 32, 11, 119 }, - { 0, 74, 111 }, - { 81, 0, 81 } - }; - if (rectangles.size() % 4 != 0 || rectangles.size() / 4 != classes.size()) { - return; - } - - for (size_t i = 0; i < classes.size(); i++) { - int x = rectangles.at(i * 4); - int y = rectangles.at(i * 4 + 1); - int w = rectangles.at(i * 4 + 2); - int h = rectangles.at(i * 4 + 3); - - int cls = classes.at(i) % colors.size(); // color of a bounding box line - - if (x < 0) x = 0; - if (y < 0) y = 0; - if (w < 0) w = 0; - if (h < 0) h = 0; - - if (static_cast(x) >= width) { x = width - 1; w = 0; thickness = 1; } - if (static_cast(y) >= height) { y = height - 1; h = 0; thickness = 1; } - - if (static_cast(x + w) >= width) { w = width - x - 1; } - if (static_cast(y + h) >= height) { h = height - y - 1; } - - thickness = std::min(std::min(thickness, w / 2 + 1), h / 2 + 1); - - size_t shift_first; - size_t shift_second; - for (int t = 0; t < thickness; t++) { - shift_first = (y + t) * width * 3; - shift_second = (y + h - t) * width * 3; - for (int ii = x; ii < x + w + 1; ii++) { - data[shift_first + ii * 3] = colors.at(cls).red(); - data[shift_first + ii * 3 + 1] = colors.at(cls).green(); - data[shift_first + ii * 3 + 2] = colors.at(cls).blue(); - data[shift_second + ii * 3] = colors.at(cls).red(); - data[shift_second + ii * 3 + 1] = colors.at(cls).green(); - data[shift_second + ii * 3 + 2] = colors.at(cls).blue(); - } - } - - for (int t = 0; t < thickness; t++) { - shift_first = (x + t) * 3; - shift_second = (x + w - t) * 3; - for (int ii = y; ii < y + h + 1; ii++) { - data[shift_first + ii * width * 3] = colors.at(cls).red(); - data[shift_first + ii * width * 3 + 1] = colors.at(cls).green(); - data[shift_first + ii * width * 3 + 2] = colors.at(cls).blue(); - data[shift_second + ii * width * 3] = colors.at(cls).red(); - data[shift_second + ii * width * 3 + 1] = colors.at(cls).green(); - data[shift_second + ii * width * 3 + 2] = colors.at(cls).blue(); - } - } - } -} - - - -/** - * Write output data to image - * \param name - image name - * \param data - output data - * \param classesNum - the number of classes - * \return false if error else true - */ - -static UNUSED bool writeOutputBmp(unsigned char *data, size_t height, size_t width, std::ostream &outFile) { - unsigned char file[14] = { - 'B', 'M', // magic - 0, 0, 0, 0, // size in bytes - 0, 0, // app data - 0, 0, // app data - 40+14, 0, 0, 0 // start of data offset - }; - unsigned char info[40] = { - 40, 0, 0, 0, // info hd size - 0, 0, 0, 0, // width - 0, 0, 0, 0, // height - 1, 0, // number color planes - 24, 0, // bits per pixel - 0, 0, 0, 0, // compression is none - 0, 0, 0, 0, // image bits size - 0x13, 0x0B, 0, 0, // horz resolution in pixel / m - 0x13, 0x0B, 0, 0, // vert resolution (0x03C3 = 96 dpi, 0x0B13 = 72 dpi) - 0, 0, 0, 0, // #colors in palette - 0, 0, 0, 0, // #important colors - }; - - if (height > (size_t)std::numeric_limits::max || width > (size_t)std::numeric_limits::max) { - THROW_IE_EXCEPTION << "File size is too big: " << height << " X " << width; - } - - int padSize = static_cast(4 - (width * 3) % 4) % 4; - int sizeData = static_cast(width * height * 3 + height * padSize); - int sizeAll = sizeData + sizeof(file) + sizeof(info); - - file[ 2] = (unsigned char)(sizeAll ); - file[ 3] = (unsigned char)(sizeAll >> 8); - file[ 4] = (unsigned char)(sizeAll >> 16); - file[ 5] = (unsigned char)(sizeAll >> 24); - - info[ 4] = (unsigned char)(width ); - info[ 5] = (unsigned char)(width >> 8); - info[ 6] = (unsigned char)(width >> 16); - info[ 7] = (unsigned char)(width >> 24); - - int32_t negativeHeight = -(int32_t)height; - info[ 8] = (unsigned char)(negativeHeight ); - info[ 9] = (unsigned char)(negativeHeight >> 8); - info[10] = (unsigned char)(negativeHeight >> 16); - info[11] = (unsigned char)(negativeHeight >> 24); - - info[20] = (unsigned char)(sizeData ); - info[21] = (unsigned char)(sizeData >> 8); - info[22] = (unsigned char)(sizeData >> 16); - info[23] = (unsigned char)(sizeData >> 24); - - outFile.write(reinterpret_cast(file), sizeof(file)); - outFile.write(reinterpret_cast(info), sizeof(info)); - - unsigned char pad[3] = {0, 0, 0}; - - for (size_t y = 0; y < height; y++) { - for (size_t x = 0; x < width; x++) { - unsigned char pixel[3]; - pixel[0] = data[y*width*3 + x*3]; - pixel[1] = data[y*width*3 + x*3 + 1]; - pixel[2] = data[y*width*3 + x*3 + 2]; - outFile.write(reinterpret_cast(pixel), 3); - } - outFile.write(reinterpret_cast(pad), padSize); - } - - return true; -} +// Known colors for training classes from the Cityscapes dataset +static UNUSED const Color CITYSCAPES_COLORS[] = { + { 128, 64, 128 }, + { 232, 35, 244 }, + { 70, 70, 70 }, + { 156, 102, 102 }, + { 153, 153, 190 }, + { 153, 153, 153 }, + { 30, 170, 250 }, + { 0, 220, 220 }, + { 35, 142, 107 }, + { 152, 251, 152 }, + { 180, 130, 70 }, + { 60, 20, 220 }, + { 0, 0, 255 }, + { 142, 0, 0 }, + { 70, 0, 0 }, + { 100, 60, 0 }, + { 90, 0, 0 }, + { 230, 0, 0 }, + { 32, 11, 119 }, + { 0, 74, 111 }, + { 81, 0, 81 } +}; static std::vector> perfCountersSorted(std::map perfMap) { @@ -581,7 +157,7 @@ perfCountersSorted(std::map& performanceMap, - std::ostream &stream, std::string deviceName, + std::ostream &stream, const std::string &deviceName, bool bshowHeader = true) { long long totalTime = 0; // Print performance counts @@ -668,353 +244,6 @@ inline std::string getFullDeviceName(InferenceEngine::Core& ie, std::string devi } } -/** - * @brief This class represents an object that is found by an object detection net - */ -class DetectedObject { -public: - int objectType; - float xmin, xmax, ymin, ymax, prob; - bool difficult; - - DetectedObject(int _objectType, float _xmin, float _ymin, float _xmax, float _ymax, float _prob, bool _difficult = false) - : objectType(_objectType), xmin(_xmin), xmax(_xmax), ymin(_ymin), ymax(_ymax), prob(_prob), difficult(_difficult) { - } - - DetectedObject(const DetectedObject& other) = default; - - static float ioU(const DetectedObject& detectedObject1_, const DetectedObject& detectedObject2_) { - // Add small space to eliminate empty squares - float epsilon = 0; // 1e-5f; - - DetectedObject detectedObject1(detectedObject1_.objectType, - (detectedObject1_.xmin - epsilon), - (detectedObject1_.ymin - epsilon), - (detectedObject1_.xmax- epsilon), - (detectedObject1_.ymax- epsilon), detectedObject1_.prob); - DetectedObject detectedObject2(detectedObject2_.objectType, - (detectedObject2_.xmin + epsilon), - (detectedObject2_.ymin + epsilon), - (detectedObject2_.xmax), - (detectedObject2_.ymax), detectedObject2_.prob); - - if (detectedObject1.objectType != detectedObject2.objectType) { - // objects are different, so the result is 0 - return 0.0f; - } - - if (detectedObject1.xmax < detectedObject1.xmin) return 0.0; - if (detectedObject1.ymax < detectedObject1.ymin) return 0.0; - if (detectedObject2.xmax < detectedObject2.xmin) return 0.0; - if (detectedObject2.ymax < detectedObject2.ymin) return 0.0; - - - float xmin = (std::max)(detectedObject1.xmin, detectedObject2.xmin); - float ymin = (std::max)(detectedObject1.ymin, detectedObject2.ymin); - float xmax = (std::min)(detectedObject1.xmax, detectedObject2.xmax); - float ymax = (std::min)(detectedObject1.ymax, detectedObject2.ymax); - - // Caffe adds 1 to every length if the box isn't normalized. So do we... - float addendum; - if (xmax > 1 || ymax > 1) - addendum = 1; - else - addendum = 0; - - // intersection - float intr; - if ((xmax >= xmin) && (ymax >= ymin)) { - intr = (addendum + xmax - xmin) * (addendum + ymax - ymin); - } else { - intr = 0.0f; - } - - // union - float square1 = (addendum + detectedObject1.xmax - detectedObject1.xmin) * (addendum + detectedObject1.ymax - detectedObject1.ymin); - float square2 = (addendum + detectedObject2.xmax - detectedObject2.xmin) * (addendum + detectedObject2.ymax - detectedObject2.ymin); - - float unn = square1 + square2 - intr; - - return static_cast(intr) / unn; - } - - DetectedObject scale(float scale_x, float scale_y) const { - return DetectedObject(objectType, xmin * scale_x, ymin * scale_y, xmax * scale_x, ymax * scale_y, prob, difficult); - } -}; - -class ImageDescription { -public: - const std::list alist; - const bool check_probs; - - explicit ImageDescription(const std::list &_alist, bool _check_probs = false) - : alist(_alist), check_probs(_check_probs) { - } - - static float ioUMultiple(const ImageDescription &detectedObjects, const ImageDescription &desiredObjects) { - const ImageDescription *detectedObjectsSmall, *detectedObjectsBig; - bool check_probs = desiredObjects.check_probs; - - if (detectedObjects.alist.size() < desiredObjects.alist.size()) { - detectedObjectsSmall = &detectedObjects; - detectedObjectsBig = &desiredObjects; - } else { - detectedObjectsSmall = &desiredObjects; - detectedObjectsBig = &detectedObjects; - } - - std::list doS = detectedObjectsSmall->alist; - std::list doB = detectedObjectsBig->alist; - - float fullScore = 0.0f; - while (doS.size() > 0) { - float score = 0.0f; - std::list::iterator bestJ = doB.end(); - for (auto j = doB.begin(); j != doB.end(); j++) { - float curscore = DetectedObject::ioU(*doS.begin(), *j); - if (score < curscore) { - score = curscore; - bestJ = j; - } - } - - float coeff = 1.0; - if (check_probs) { - if (bestJ != doB.end()) { - float mn = std::min((*bestJ).prob, (*doS.begin()).prob); - float mx = std::max((*bestJ).prob, (*doS.begin()).prob); - - coeff = mn/mx; - } - } - - doS.pop_front(); - if (bestJ != doB.end()) doB.erase(bestJ); - fullScore += coeff * score; - } - fullScore /= detectedObjectsBig->alist.size(); - - - return fullScore; - } - - ImageDescription scale(float scale_x, float scale_y) const { - std::list slist; - for (auto& dob : alist) { - slist.push_back(dob.scale(scale_x, scale_y)); - } - return ImageDescription(slist, check_probs); - } -}; - -struct AveragePrecisionCalculator { -private: - enum MatchKind { - TruePositive, FalsePositive - }; - - /** - * Here we count all TP and FP matches for all the classes in all the images. - */ - std::map>> matches; - - std::map N; - - double threshold; - - static bool SortBBoxDescend(const DetectedObject& bbox1, const DetectedObject& bbox2) { - return bbox1.prob > bbox2.prob; - } - - static bool SortPairDescend(const std::pair& p1, const std::pair& p2) { - return p1.first > p2.first; - } - -public: - explicit AveragePrecisionCalculator(double _threshold) : threshold(_threshold) { } - - // gt_bboxes -> des - // bboxes -> det - - void consumeImage(const ImageDescription &detectedObjects, const ImageDescription &desiredObjects) { - // Collecting IoU values - std::vector visited(desiredObjects.alist.size(), false); - std::vector bboxes{ std::begin(detectedObjects.alist), std::end(detectedObjects.alist) }; - std::sort(bboxes.begin(), bboxes.end(), SortBBoxDescend); - - - for (auto&& detObj : bboxes) { - // Searching for the best match to this detection - // Searching for desired object - float overlap_max = -1; - int jmax = -1; - auto desmax = desiredObjects.alist.end(); - - int j = 0; - for (auto desObj = desiredObjects.alist.begin(); desObj != desiredObjects.alist.end(); desObj++, j++) { - double iou = DetectedObject::ioU(detObj, *desObj); - if (iou > overlap_max) { - overlap_max = static_cast(iou); - jmax = j; - desmax = desObj; - } - } - - MatchKind mk; - if (overlap_max >= threshold) { - if (!desmax->difficult) { - if (!visited[jmax]) { - mk = TruePositive; - visited[jmax] = true; - } else { - mk = FalsePositive; - } - matches[detObj.objectType].push_back(std::make_pair(detObj.prob, mk)); - } - } else { - mk = FalsePositive; - matches[detObj.objectType].push_back(std::make_pair(detObj.prob, mk)); - } - } - - for (auto desObj = desiredObjects.alist.begin(); desObj != desiredObjects.alist.end(); desObj++) { - if (!desObj->difficult) { - N[desObj->objectType]++; - } - } - } - - std::map calculateAveragePrecisionPerClass() const { - /** - * Precision-to-TP curve per class (a variation of precision-to-recall curve without dividing into N) - */ - std::map> precisionToTP; - - - std::map res; - - for (auto m : matches) { - // Sorting - std::sort(m.second.begin(), m.second.end(), SortPairDescend); - - int clazz = m.first; - int TP = 0, FP = 0; - - std::vector prec; - std::vector rec; - - for (auto mm : m.second) { - // Here we are descending in a probability value - MatchKind mk = mm.second; - if (mk == TruePositive) TP++; - else if (mk == FalsePositive) FP++; - - double precision = static_cast(TP) / (TP + FP); - double recall = 0; - if (N.find(clazz) != N.end()) { - recall = static_cast(TP) / N.at(clazz); - } - - prec.push_back(precision); - rec.push_back(recall); - } - - int num = rec.size(); - - // 11point from Caffe - double ap = 0; - std::vector max_precs(11, 0.); - int start_idx = num - 1; - for (int j = 10; j >= 0; --j) { - for (int i = start_idx; i >= 0; --i) { - if (rec[i] < j / 10.) { - start_idx = i; - if (j > 0) { - max_precs[j-1] = max_precs[j]; - } - break; - } else { - if (max_precs[j] < prec[i]) { - max_precs[j] = static_cast(prec[i]); - } - } - } - } - for (int j = 10; j >= 0; --j) { - ap += max_precs[j] / 11; - } - res[clazz] = ap; - } - - return res; - } -}; - -/** -* @brief Adds colored rectangles to the image -* @param data - data where rectangles are put -* @param height - height of the rectangle -* @param width - width of the rectangle -* @param detectedObjects - vector of detected objects -*/ -static UNUSED void addRectangles(unsigned char *data, size_t height, size_t width, std::vector detectedObjects) { - std::vector colors = { - { 128, 64, 128 }, - { 232, 35, 244 }, - { 70, 70, 70 }, - { 156, 102, 102 }, - { 153, 153, 190 }, - { 153, 153, 153 }, - { 30, 170, 250 }, - { 0, 220, 220 }, - { 35, 142, 107 }, - { 152, 251, 152 }, - { 180, 130, 70 }, - { 60, 20, 220 }, - { 0, 0, 255 }, - { 142, 0, 0 }, - { 70, 0, 0 }, - { 100, 60, 0 }, - { 90, 0, 0 }, - { 230, 0, 0 }, - { 32, 11, 119 }, - { 0, 74, 111 }, - { 81, 0, 81 } - }; - - for (size_t i = 0; i < detectedObjects.size(); i++) { - int cls = detectedObjects[i].objectType % colors.size(); - - int xmin = static_cast(detectedObjects[i].xmin * width); - int xmax = static_cast(detectedObjects[i].xmax * width); - int ymin = static_cast(detectedObjects[i].ymin * height); - int ymax = static_cast(detectedObjects[i].ymax * height); - - size_t shift_first = ymin*width * 3; - size_t shift_second = ymax*width * 3; - for (int x = xmin; x < xmax; x++) { - data[shift_first + x * 3] = colors.at(cls).red(); - data[shift_first + x * 3 + 1] = colors.at(cls).green(); - data[shift_first + x * 3 + 2] = colors.at(cls).blue(); - data[shift_second + x * 3] = colors.at(cls).red(); - data[shift_second + x * 3 + 1] = colors.at(cls).green(); - data[shift_second + x * 3 + 2] = colors.at(cls).blue(); - } - - shift_first = xmin * 3; - shift_second = xmax * 3; - for (int y = ymin; y < ymax; y++) { - data[shift_first + y*width * 3] = colors.at(cls).red(); - data[shift_first + y*width * 3 + 1] = colors.at(cls).green(); - data[shift_first + y*width * 3 + 2] = colors.at(cls).blue(); - data[shift_second + y*width * 3] = colors.at(cls).red(); - data[shift_second + y*width * 3 + 1] = colors.at(cls).green(); - data[shift_second + y*width * 3 + 2] = colors.at(cls).blue(); - } - } -} - inline std::size_t getTensorWidth(const InferenceEngine::TensorDesc& desc) { const auto& layout = desc.getLayout(); const auto& dims = desc.getDims(); @@ -1120,4 +349,5 @@ inline void showAvailableDevices() { for (const auto& device : devices) { std::cout << " " << device; } + std::cout << std::endl; } diff --git a/demos/common/samples/console_progress.hpp b/demos/common/samples/console_progress.hpp deleted file mode 100644 index 5edfea80fbc..00000000000 --- a/demos/common/samples/console_progress.hpp +++ /dev/null @@ -1,90 +0,0 @@ -// Copyright (C) 2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#pragma once - -#include -#include - -/** - * @class ConsoleProgress - * @brief A ConsoleProgress class provides functionality for printing progress dynamics - */ -class ConsoleProgress { - static const int DEFAULT_DETALIZATION = 20; - - size_t total; - size_t current = 0; - bool stream_output; - size_t detalization; - -public: - /** - * @brief A constructor of ConsoleProgress class - * @param _total - maximum value that is correspondent to 100% - * @param _detalization - number of symbols(.) to use to represent progress - */ - explicit ConsoleProgress(size_t _total, bool _stream_output = false, size_t _detalization = DEFAULT_DETALIZATION) : - total(_total), detalization(_detalization) { - stream_output = _stream_output; - if (total == 0) { - total = 1; - } - std::cout << std::unitbuf; - } - - /** - * @brief Shows progress with current data. Progress is shown from the beginning of the current line. - * @return - */ - void showProgress() const { - std::stringstream strm; - if (!stream_output) { - strm << '\r'; - } - strm << "Progress: ["; - size_t i = 0; - for (; i < detalization * current / total; i++) { - strm << "."; - } - for (; i < detalization; i++) { - strm << " "; - } - strm << "] " << std::fixed << std::setprecision(2) << 100 * static_cast(current) / total << "% done"; - if (stream_output) { - std::cout << strm.str() << std::endl; - } else { - std::cout << strm.str() << std::flush; - } - } - - /** - * @brief Updates current value and progressbar - * @param newProgress - new value to represent - */ - void updateProgress(size_t newProgress) { - current = newProgress; - if (current > total) current = total; - showProgress(); - } - - /** - * @brief Adds value to currently represented and redraw progressbar - * @param add - value to add - */ - void addProgress(int add) { - if (add < 0 && -add > static_cast(current)) { - add = -static_cast(current); - } - updateProgress(current + add); - } - - /** - * @brief Output end line. - * @return - */ - void finish() { - std::cerr << std::nounitbuf << "\n"; - } -}; diff --git a/demos/common/samples/csv_dumper.hpp b/demos/common/samples/csv_dumper.hpp deleted file mode 100644 index 4dbcfa19f38..00000000000 --- a/demos/common/samples/csv_dumper.hpp +++ /dev/null @@ -1,98 +0,0 @@ -// Copyright (C) 2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#pragma once - -#include -#include -#include -#include -#include - -#include - -/** - * @class CsvDumper - * @brief A CsvDumper class provides functionality for dumping the values in CSV files - */ -class CsvDumper { - std::ofstream file; - std::string filename; - bool canDump = true; - char delimiter = ';'; - - std::string generateFilename() { - std::stringstream filename; - filename << "dumpfile-"; - filename << time(nullptr); - filename << ".csv"; - return filename.str(); - } - -public: - /** - * @brief A constructor. Disables dumping in case dump file cannot be created - * @param enabled - True if dumping is enabled by default. - * @param name - name of file to dump to. File won't be created if first parameter is false. - */ - explicit CsvDumper(bool enabled = true, const std::string& name = "") : canDump(enabled) { - if (!canDump) { - return; - } - filename = (name == "" ? generateFilename() : name); - file.open(filename, std::ios::out); - if (!file) { - slog::warn << "Cannot create dump file! Disabling dump." << slog::endl; - canDump = false; - } - } - - /** - * @brief Sets a delimiter to use in csv file - * @param c - Delimiter char - * @return - */ - void setDelimiter(char c) { - delimiter = c; - } - - /** - * @brief Overloads operator to organize streaming values to file. Does nothing if dumping is disabled - * Adds delimiter at the end of value provided - * @param add - value to add to dump - * @return reference to same object - */ - template - CsvDumper& operator<<(const T& add) { - if (canDump) { - file << add << delimiter; - } - return *this; - } - - /** - * @brief Finishes line in dump file. Does nothing if dumping is disabled - */ - void endLine() { - if (canDump) { - file << "\n"; - } - } - - /** - * @brief Gets information if dump is enabled. - * @return true if dump is enabled and file was successfully created - */ - bool dumpEnabled() { - return canDump; - } - - /** - * @brief Gets name of a dump file - * @return name of a dump file - */ - std::string getFilename() const { - return filename; - } -}; diff --git a/demos/common/samples/ocv_common.hpp b/demos/common/samples/ocv_common.hpp index 770b0d7df75..ee29eca0f1c 100644 --- a/demos/common/samples/ocv_common.hpp +++ b/demos/common/samples/ocv_common.hpp @@ -24,6 +24,9 @@ void matU8ToBlob(const cv::Mat& orig_image, InferenceEngine::Blob::Ptr& blob, in const size_t width = blobSize[3]; const size_t height = blobSize[2]; const size_t channels = blobSize[1]; + if (static_cast(orig_image.channels()) != channels) { + THROW_IE_EXCEPTION << "The number of channels for net input and image must match"; + } T* blob_data = blob->buffer().as(); cv::Mat resized_image(orig_image); diff --git a/demos/common/samples/slog.hpp b/demos/common/samples/slog.hpp index 186c2cf0a6d..f3b34f4e95f 100644 --- a/demos/common/samples/slog.hpp +++ b/demos/common/samples/slog.hpp @@ -42,7 +42,7 @@ class LogStream { public: /** - * @brief A constructor. Creates an LogStream object + * @brief A constructor. Creates a LogStream object * @param prefix The prefix to print */ LogStream(const std::string &prefix, std::ostream& log_stream) diff --git a/demos/common/vpu/vpu_tools_common.hpp b/demos/common/vpu/vpu_tools_common.hpp deleted file mode 100644 index e2185a8b955..00000000000 --- a/demos/common/vpu/vpu_tools_common.hpp +++ /dev/null @@ -1,27 +0,0 @@ -// Copyright (C) 2019 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#pragma once - -#include -#include - -static std::map parseConfig(const std::string &configName, char comment = '#') { - std::map config = {}; - - std::ifstream file(configName); - if (!file.is_open()) { - return config; - } - - std::string key, value; - while (file >> key >> value) { - if (key.empty() || key[0] == comment) { - continue; - } - config[key] = value; - } - - return config; -} diff --git a/demos/crossroad_camera_demo/CMakeLists.txt b/demos/crossroad_camera_demo/CMakeLists.txt index 0ba0f10f725..d3c76ab36ba 100644 --- a/demos/crossroad_camera_demo/CMakeLists.txt +++ b/demos/crossroad_camera_demo/CMakeLists.txt @@ -5,4 +5,5 @@ ie_add_sample(NAME crossroad_camera_demo SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/main.cpp" HEADERS "${CMAKE_CURRENT_SOURCE_DIR}/crossroad_camera_demo.hpp" + DEPENDENCIES monitors OPENCV_DEPENDENCIES highgui) diff --git a/demos/crossroad_camera_demo/README.md b/demos/crossroad_camera_demo/README.md index e5964695c37..300986acb25 100644 --- a/demos/crossroad_camera_demo/README.md +++ b/demos/crossroad_camera_demo/README.md @@ -5,7 +5,7 @@ This demo provides an inference pipeline for persons' detection, recognition and * `person-vehicle-bike-detection-crossroad-0078`, which is a primary detection network for finding the persons (and other objects if needed) * `person-attributes-recognition-crossroad-0230`, which is executed on top of the results from the first network and reports person attributes like gender, has hat, has long-sleeved clothes -* `person-reidentification-retail-0079`, which is executed on top of the results from the first network and prints +* `person-reidentification-retail-0031`, which is executed on top of the results from the first network and prints a vector of features for each detected person. This vector is used to conclude if it is already detected person or not. For more information about the pre-trained models, refer to the [model documentation](../../models/intel/index.md). @@ -36,7 +36,7 @@ REID value is assigned. Otherwise, the vector is added to a global list, and new ## Running Running the application with the `-h` option yields the following usage message: -```sh +``` ./crossroad_camera_demo -h InferenceEngine: API version ............ @@ -62,6 +62,7 @@ Options: -t_reid Optional. Cosine similarity threshold between two vectors for person reidentification. -no_show Optional. No show processed video. -auto_resize Optional. Enables resizable input with support of ROI crop & auto resize. + -u Optional. List of monitors to show initially. ``` Running the application with an empty list of options yields the usage message given above and an error message. @@ -87,7 +88,7 @@ If Person Attributes Recognition or Person Reidentification Retail are enabled, > **NOTE**: On VPU devices (Intel® Movidius™ Neural Compute Stick, Intel® Neural Compute Stick 2, and Intel® Vision Accelerator Design with Intel® Movidius™ VPUs) this demo has been tested on the following Model Downloader available topologies: >* `person-attributes-recognition-crossroad-0230` ->* `person-reidentification-retail-0079` +>* `person-reidentification-retail-0031` >* `person-vehicle-bike-detection-crossroad-0078` > Other models may produce unexpected results on these devices. diff --git a/demos/crossroad_camera_demo/crossroad_camera_demo.hpp b/demos/crossroad_camera_demo/crossroad_camera_demo.hpp index d5fcf5973af..184b25b3a6a 100644 --- a/demos/crossroad_camera_demo/crossroad_camera_demo.hpp +++ b/demos/crossroad_camera_demo/crossroad_camera_demo.hpp @@ -9,120 +9,58 @@ #include #include -/// @brief message for help argument static const char help_message[] = "Print a usage message."; - -/// @brief message for images argument static const char video_message[] = "Required. Path to a video or image file. Default value is \"cam\" to work with camera."; - -/// @brief message for model argument static const char person_vehicle_bike_detection_model_message[] = "Required. Path to the Person/Vehicle/Bike Detection Crossroad model (.xml) file."; static const char person_attribs_model_message[] = "Optional. Path to the Person Attributes Recognition Crossroad model (.xml) file."; static const char person_reid_model_message[] = "Optional. Path to the Person Reidentification Retail model (.xml) file."; - -/// @brief message for assigning Person/Vehicle/Bike detection inference to device -static const char target_device_message[] = "Optional. Specify the target device for Person/Vehicle/Bike Detection. " \ - "The list of available devices is shown below. Default value is CPU. " \ - "Use \"-d HETERO:\" format to specify HETERO plugin. " \ +static const char target_device_message[] = "Optional. Specify the target device for Person/Vehicle/Bike Detection. " + "The list of available devices is shown below. Default value is CPU. " + "Use \"-d HETERO:\" format to specify HETERO plugin. " "The application looks for a suitable plugin for the specified device."; - -/// @brief message for assigning Person attributes recognition inference to device -static const char target_device_message_person_attribs[] = "Optional. Specify the target device for Person Attributes Recognition. "\ - "The list of available devices is shown below. Default value is CPU. " \ - "Use \"-d HETERO:\" format to specify HETERO plugin. " \ +static const char target_device_message_person_attribs[] = "Optional. Specify the target device for Person Attributes Recognition. " + "The list of available devices is shown below. Default value is CPU. " + "Use \"-d HETERO:\" format to specify HETERO plugin. " "The application looks for a suitable plugin for the specified device."; - -/// @brief message for assigning Person Reidentification retail inference to device -static const char target_device_message_person_reid[] = "Optional. Specify the target device for Person Reidentification Retail. "\ - "The list of available devices is shown below. Default value is CPU. " \ - "Use \"-d HETERO:\" format to specify HETERO plugin. " \ +static const char target_device_message_person_reid[] = "Optional. Specify the target device for Person Reidentification Retail. " + "The list of available devices is shown below. Default value is CPU. " + "Use \"-d HETERO:\" format to specify HETERO plugin. " "The application looks for a suitable plugin for the specified device."; - -/// @brief message for performance counters static const char performance_counter_message[] = "Optional. Enables per-layer performance statistics."; - -/// @brief message for clDNN custom kernels desc -static const char custom_cldnn_message[] = "Optional. For clDNN (GPU)-targeted custom kernels, if any. "\ -"Absolute path to the xml file with the kernels desc."; - -/// @brief message for user library argument -static const char custom_cpu_library_message[] = "Optional. For MKLDNN (CPU)-targeted custom layers, if any. " \ -"Absolute path to a shared library with the kernels impl."; - -/// @brief message for probability threshold argument for person/vehicle/bike crossroad detections +static const char custom_cldnn_message[] = "Optional. For clDNN (GPU)-targeted custom kernels, if any. " + "Absolute path to the xml file with the kernels desc."; +static const char custom_cpu_library_message[] = "Optional. For MKLDNN (CPU)-targeted custom layers, if any. " + "Absolute path to a shared library with the kernels impl."; static const char threshold_output_message[] = "Optional. Probability threshold for person/vehicle/bike crossroad detections."; - -/// @brief message for probability threshold argument for person/vehicle/bike crossroad detections static const char threshold_output_message_person_reid[] = "Optional. Cosine similarity threshold between two vectors for person reidentification."; - -/// @brief message raw output flag static const char raw_output_message[] = "Optional. Output Inference results as raw values."; - -/// @brief message no show processed video static const char no_show_processed_video[] = "Optional. No show processed video."; - -/// @brief message resizable input flag static const char input_resizable_message[] = "Optional. Enables resizable input with support of ROI crop & auto resize."; +/// @brief Message list of monitors to show +static const char utilization_monitors_message[] = "Optional. List of monitors to show initially."; -/// @brief Define flag for showing help message
-DEFINE_bool(h, false, help_message); -/// @brief Define parameter for set image file
-/// It is a required parameter +DEFINE_bool(h, false, help_message); DEFINE_string(i, "cam", video_message); - -/// @brief Define parameter for vehicle detection model file
-/// It is a required parameter DEFINE_string(m, "", person_vehicle_bike_detection_model_message); - -/// @brief Define parameter for vehicle attributes model file
-/// It is a required parameter DEFINE_string(m_pa, "", person_attribs_model_message); - -/// @brief Define parameter for vehicle detection model file
-/// It is a required parameter DEFINE_string(m_reid, "", person_reid_model_message); - -/// @brief device the target device for vehicle detection infer on
DEFINE_string(d, "CPU", target_device_message); - -/// @brief device the target device for age gender detection on
DEFINE_string(d_pa, "CPU", target_device_message_person_attribs); - -/// @brief device the target device for head pose detection on
DEFINE_string(d_reid, "CPU", target_device_message_person_reid); - -/// @brief Enable per-layer performance report DEFINE_bool(pc, false, performance_counter_message); - -/// @brief clDNN custom kernels path
-/// Default is ./lib DEFINE_string(c, "", custom_cldnn_message); - -/// @brief Absolute path to CPU library with user layers
-/// It is a optional parameter DEFINE_string(l, "", custom_cpu_library_message); - -/// @brief Flag to output raw scoring results
-/// It is an optional parameter DEFINE_bool(r, false, raw_output_message); - -/// @brief Define probability threshold for person/vehicle/bike crossroad detections
-/// It is an optional parameter DEFINE_double(t, 0.5, threshold_output_message); - -/// @brief Define probability threshold for person/vehicle/bike crossroad detections
-/// It is an optional parameter DEFINE_double(t_reid, 0.7, threshold_output_message_person_reid); - -/// @brief Flag to disable processed video showing
-/// It is an optional parameter DEFINE_bool(no_show, false, no_show_processed_video); +DEFINE_bool(auto_resize, false, input_resizable_message); -/// \brief Enables resizable input
+/// \brief Define a flag to show monitors
/// It is an optional parameter -DEFINE_bool(auto_resize, false, input_resizable_message); +DEFINE_string(u, "", utilization_monitors_message); /** @@ -150,4 +88,5 @@ static void showUsage() { std::cout << " -t_reid " << threshold_output_message_person_reid << std::endl; std::cout << " -no_show " << no_show_processed_video << std::endl; std::cout << " -auto_resize " << input_resizable_message << std::endl; + std::cout << " -u " << utilization_monitors_message << std::endl; } diff --git a/demos/crossroad_camera_demo/main.cpp b/demos/crossroad_camera_demo/main.cpp index 82bd63b69ba..cd83821a38c 100644 --- a/demos/crossroad_camera_demo/main.cpp +++ b/demos/crossroad_camera_demo/main.cpp @@ -23,12 +23,10 @@ #include +#include #include #include #include "crossroad_camera_demo.hpp" -#ifdef WITH_EXTENSIONS -#include -#endif using namespace InferenceEngine; @@ -66,13 +64,13 @@ struct BaseDetection { std::string inputName; std::string outputName; - BaseDetection(std::string &commandLineFlag, std::string topoName) + BaseDetection(std::string &commandLineFlag, const std::string &topoName) : commandLineFlag(commandLineFlag), topoName(topoName) {} ExecutableNetwork * operator ->() { return &net; } - virtual CNNNetwork read() = 0; + virtual CNNNetwork read(const Core& ie) = 0; virtual void setRoiBlob(const Blob::Ptr &roiBlob) { if (!enabled()) @@ -160,23 +158,19 @@ struct PersonDetection : BaseDetection{ } PersonDetection() : BaseDetection(FLAGS_m, "Person Detection"), maxProposalCount(0), objectSize(0) {} - CNNNetwork read() override { + CNNNetwork read(const Core& ie) override { slog::info << "Loading network files for PersonDetection" << slog::endl; - CNNNetReader netReader; /** Read network model **/ - netReader.ReadNetwork(FLAGS_m); + auto network = ie.ReadNetwork(FLAGS_m); /** Set batch size to 1 **/ slog::info << "Batch size is forced to 1" << slog::endl; - netReader.getNetwork().setBatchSize(1); - /** Extract model name and load it's weights **/ - std::string binFileName = fileNameNoExt(FLAGS_m) + ".bin"; - netReader.ReadWeights(binFileName); + network.setBatchSize(1); // ----------------------------------------------------------------------------------------------------- /** SSD-based network should have one input and one output **/ // ---------------------------Check inputs ------------------------------------------------------ slog::info << "Checking Person Detection inputs" << slog::endl; - InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Person Detection network should have only one input"); } @@ -194,7 +188,7 @@ struct PersonDetection : BaseDetection{ // ---------------------------Check outputs ------------------------------------------------------ slog::info << "Checking Person Detection outputs" << slog::endl; - OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 1) { throw std::logic_error("Person Detection network should have only one output"); } @@ -213,7 +207,7 @@ struct PersonDetection : BaseDetection{ _output->setLayout(Layout::NCHW); slog::info << "Loading Person Detection model to the "<< FLAGS_d << " device" << slog::endl; - return netReader.getNetwork(); + return network; } void fetchResults() { @@ -282,22 +276,19 @@ struct PersonAttribsDetection : BaseDetection { 10, cv::KMEANS_RANDOM_CENTERS, centers); centers.convertTo(centers, CV_8U); centers = centers.reshape(0, clusterCount); - std::map> max_color; std::vector freq(clusterCount); for (int i = 0; i < labels.rows * labels.cols; ++i) { freq[labels.at(i)]++; } - for (size_t i = 0; i < freq.size(); ++i) { - max_color[freq[i]] = centers.at(i); - } + auto freqArgmax = std::max_element(freq.begin(), freq.end()) - freq.begin(); - return max_color.begin()->second; + return centers.at(freqArgmax); } AttributesAndColorPoints GetPersonAttributes() { - static const std::vector attributesVec = { + static const char *const attributeStrings[] = { "is male", "has_bag", "has_backpack" , "has hat", "has longsleeves", "has longpants", "has longhair", "has coat_jacket" }; @@ -308,10 +299,10 @@ struct PersonAttribsDetection : BaseDetection { size_t numOfTCPointChannels = topColorPointBlob->getTensorDesc().getDims().at(1); size_t numOfBCPointChannels = bottomColorPointBlob->getTensorDesc().getDims().at(1); - if (numOfAttrChannels != attributesVec.size()) { + if (numOfAttrChannels != arraySize(attributeStrings)) { throw std::logic_error("Output size (" + std::to_string(numOfAttrChannels) + ") of the " - "Person Attributes Recognition network is not equal to used person " - "attributes vector size (" + std::to_string(attributesVec.size()) + ")"); + "Person Attributes Recognition network is not equal to expected " + "number of attributes (" + std::to_string(arraySize(attributeStrings)) + ")"); } if (numOfTCPointChannels != 2) { throw std::logic_error("Output size (" + std::to_string(numOfTCPointChannels) + ") of the " @@ -334,31 +325,27 @@ struct PersonAttribsDetection : BaseDetection { returnValue.bottom_color_point.x = outputBCPointValues[0]; returnValue.bottom_color_point.y = outputBCPointValues[1]; - for (size_t i = 0; i < attributesVec.size(); i++) { - returnValue.attributes_strings.push_back(attributesVec[i]); + for (size_t i = 0; i < arraySize(attributeStrings); i++) { + returnValue.attributes_strings.push_back(attributeStrings[i]); returnValue.attributes_indicators.push_back(outputAttrValues[i] > 0.5); } return returnValue; } - CNNNetwork read() override { + CNNNetwork read(const Core& ie) override { slog::info << "Loading network files for PersonAttribs" << slog::endl; - CNNNetReader netReader; /** Read network model **/ - netReader.ReadNetwork(FLAGS_m_pa); - netReader.getNetwork().setBatchSize(1); - slog::info << "Batch size is forced to 1 for Person Attribs" << slog::endl; - + auto network = ie.ReadNetwork(FLAGS_m_pa); /** Extract model name and load it's weights **/ - std::string binFileName = fileNameNoExt(FLAGS_m_pa) + ".bin"; - netReader.ReadWeights(binFileName); + network.setBatchSize(1); + slog::info << "Batch size is forced to 1 for Person Attribs" << slog::endl; // ----------------------------------------------------------------------------------------------------- /** Person Attribs network should have one input two outputs **/ // ---------------------------Check inputs ------------------------------------------------------ slog::info << "Checking PersonAttribs inputs" << slog::endl; - InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Person Attribs topology should have only one input"); } @@ -375,7 +362,7 @@ struct PersonAttribsDetection : BaseDetection { // ---------------------------Check outputs ------------------------------------------------------ slog::info << "Checking Person Attribs outputs" << slog::endl; - OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 3) { throw std::logic_error("Person Attribs Network expects networks having one output"); } @@ -385,7 +372,7 @@ struct PersonAttribsDetection : BaseDetection { outputNameForBottomColorPoint = (it++)->second->getName(); // bottom color location slog::info << "Loading Person Attributes Recognition model to the "<< FLAGS_d_pa << " device" << slog::endl; _enabled = true; - return netReader.getNetwork(); + return network; } }; @@ -395,12 +382,11 @@ struct PersonReIdentification : BaseDetection { PersonReIdentification() : BaseDetection(FLAGS_m_reid, "Person Reidentification Retail") {} unsigned long int findMatchingPerson(const std::vector &newReIdVec) { - float cosSim; auto size = globalReIdVec.size(); /* assigned REID is index of the matched vector from the globalReIdVec */ for (size_t i = 0; i < size; ++i) { - cosSim = cosineSimilarity(newReIdVec, globalReIdVec[i]); + float cosSim = cosineSimilarity(newReIdVec, globalReIdVec[i]); if (FLAGS_r) { std::cout << "cosineSimilarity: " << cosSim << std::endl; } @@ -419,14 +405,8 @@ struct PersonReIdentification : BaseDetection { Blob::Ptr attribsBlob = request.GetBlob(outputName); auto numOfChannels = attribsBlob->getTensorDesc().getDims().at(1); - /* output descriptor of Person Reidentification Recognition network has size 256 */ - if (numOfChannels != 256) { - throw std::logic_error("Output size (" + std::to_string(numOfChannels) + ") of the " - "Person Reidentification network is not equal to 256"); - } - auto outputValues = attribsBlob->buffer().as(); - return std::vector(outputValues, outputValues + 256); + return std::vector(outputValues, outputValues + numOfChannels); } template @@ -453,21 +433,16 @@ struct PersonReIdentification : BaseDetection { return mul / (sqrt(denomA) * sqrt(denomB)); } - CNNNetwork read() override { + CNNNetwork read(const Core& ie) override { slog::info << "Loading network files for Person Reidentification" << slog::endl; - CNNNetReader netReader; /** Read network model **/ - netReader.ReadNetwork(FLAGS_m_reid); + auto network = ie.ReadNetwork(FLAGS_m_reid); slog::info << "Batch size is forced to 1 for Person Reidentification Network" << slog::endl; - netReader.getNetwork().setBatchSize(1); - /** Extract model name and load it's weights **/ - std::string binFileName = fileNameNoExt(FLAGS_m_reid) + ".bin"; - netReader.ReadWeights(binFileName); - + network.setBatchSize(1); /** Person Reidentification network should have 1 input and one output **/ // ---------------------------Check inputs ------------------------------------------------------ slog::info << "Checking Person Reidentification Network input" << slog::endl; - InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Person Reidentification Retail should have 1 input"); } @@ -484,7 +459,7 @@ struct PersonReIdentification : BaseDetection { // ---------------------------Check outputs ------------------------------------------------------ slog::info << "Checking Person Reidentification Network output" << slog::endl; - OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 1) { throw std::logic_error("Person Reidentification Network should have 1 output"); } @@ -492,7 +467,7 @@ struct PersonReIdentification : BaseDetection { slog::info << "Loading Person Reidentification Retail model to the "<< FLAGS_d_reid << " device" << slog::endl; _enabled = true; - return netReader.getNetwork(); + return network; } }; @@ -502,7 +477,7 @@ struct Load { void into(Core & ie, const std::string & deviceName) const { if (detector.enabled()) { - detector.net = ie.LoadNetwork(detector.read(), deviceName); + detector.net = ie.LoadNetwork(detector.read(ie), deviceName); } } }; @@ -559,10 +534,6 @@ int main(int argc, char *argv[]) { std::cout << ie.GetVersions(flag) << std::endl; if ((flag.find("CPU") != std::string::npos)) { -#ifdef WITH_EXTENSIONS - /** Load default extensions lib for the CPU device (e.g. SSD's DetectionOutput)**/ - ie.AddExtension(std::make_shared(), "CPU"); -#endif if (!FLAGS_l.empty()) { // CPU(MKLDNN) extensions are loaded as a shared library and passed as a pointer to base extension auto extension_ptr = make_so_pointer(FLAGS_l); @@ -608,6 +579,9 @@ int main(int argc, char *argv[]) { } std::cout << std::endl; + cv::Size graphSize{static_cast(cap.get(cv::CAP_PROP_FRAME_WIDTH) / 4), 60}; + Presenter presenter(FLAGS_u, static_cast(cap.get(cv::CAP_PROP_FRAME_HEIGHT)) - graphSize.height - 10, graphSize); + do { // get and enqueue the next frame (in case of video) if (isVideo && !cap.read(frame)) { @@ -780,6 +754,8 @@ int main(int argc, char *argv[]) { } } + presenter.drawGraphs(frame); + // --------------------------- Execution statistics ------------------------------------------------ std::ostringstream out; out << "Person detection time : " << std::fixed << std::setprecision(2) << detection.count() @@ -820,6 +796,7 @@ int main(int argc, char *argv[]) { const int key = cv::waitKey(isVideo ? 1 : 0); if (27 == key) // Esc break; + presenter.handleKey(key); } } while (isVideo); @@ -843,6 +820,8 @@ int main(int argc, char *argv[]) { personReId.printPerformanceCounts(getFullDeviceName(mapDevices, FLAGS_d_reid)); } } + + std::cout << presenter.reportMeans() << '\n'; // ----------------------------------------------------------------------------------------------------- } catch (const std::exception& error) { diff --git a/demos/gaze_estimation_demo/CMakeLists.txt b/demos/gaze_estimation_demo/CMakeLists.txt index 02f7d4e0459..deece6f776b 100644 --- a/demos/gaze_estimation_demo/CMakeLists.txt +++ b/demos/gaze_estimation_demo/CMakeLists.txt @@ -9,4 +9,5 @@ ie_add_sample(NAME gaze_estimation_demo SOURCES ${SOURCES} HEADERS ${HEADERS} INCLUDE_DIRECTORIES "${CMAKE_CURRENT_SOURCE_DIR}/include" + DEPENDENCIES monitors OPENCV_DEPENDENCIES highgui) diff --git a/demos/gaze_estimation_demo/README.md b/demos/gaze_estimation_demo/README.md index e6d5d594128..ed49233bd54 100644 --- a/demos/gaze_estimation_demo/README.md +++ b/demos/gaze_estimation_demo/README.md @@ -27,7 +27,7 @@ Other demo objectives are: ## Running Running the application with the `-h` option yields the following usage message: -```sh +``` ./gaze_estimation_demo -h InferenceEngine: API version ............ @@ -52,6 +52,7 @@ Options: -pc Optional. Enable per-layer performance report. -r Optional. Output inference results as raw values. -t Optional. Probability threshold for Face Detector. The default value is 0.5. + -u Optional. List of monitors to show initially. ``` Running the application with an empty list of options yields an error message. diff --git a/demos/gaze_estimation_demo/gaze_estimation_demo.hpp b/demos/gaze_estimation_demo/gaze_estimation_demo.hpp index fac7ee02444..eadacdfe12d 100644 --- a/demos/gaze_estimation_demo/gaze_estimation_demo.hpp +++ b/demos/gaze_estimation_demo/gaze_estimation_demo.hpp @@ -16,118 +16,51 @@ #include #endif -/// @brief Message for help argument static const char help_message[] = "Print a usage message."; - -/// @brief Message for images argument static const char video_message[] = "Optional. Path to a video file. Default value is \"cam\" to work with camera."; - -/// @brief message for model argument static const char gaze_estimation_model_message[] = "Required. Path to an .xml file with a trained Gaze Estimation model."; static const char face_detection_model_message[] = "Required. Path to an .xml file with a trained Face Detection model."; static const char head_pose_model_message[] = "Required. Path to an .xml file with a trained Head Pose Estimation model."; static const char facial_landmarks_model_message[] = "Required. Path to an .xml file with a trained Facial Landmarks Estimation model."; - -/// @brief Message for plugin argument -static const char plugin_message[] = "Plugin name. For example, CPU. If this parameter is specified, " \ -"the demo will look for this plugin only."; - -/// @brief Message for assigning gaze calculation to device -static const char target_device_message[] = "Optional. Target device for Gaze Estimation network (the list of available devices is shown below). " \ -"Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device. Default value is \"CPU\"."; - -/// @brief Message for assigning face detection calculation to device -static const char target_device_message_fd[] = "Optional. Target device for Face Detection network (the list of available devices is shown below). " \ -"Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device. Default value is \"CPU\"."; - -/// @brief Message for assigning head pose calculation to device -static const char target_device_message_hp[] = "Optional. Target device for Head Pose Estimation network (the list of available devices is shown below). " \ -"Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device. Default value is \"CPU\"."; - -/// @brief Message for assigning facial landmarks calculation to device -static const char target_device_message_lm[] = "Optional. Target device for Facial Landmarks Estimation network " \ -"(the list of available devices is shown below). Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for device specified. Default value is \"CPU\"."; - -/// @brief Message for assigning setting resolution of camera +static const char plugin_message[] = "Plugin name. For example, CPU. If this parameter is specified, " + "the demo will look for this plugin only."; +static const char target_device_message[] = "Optional. Target device for Gaze Estimation network (the list of available devices is shown below). " + "Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device. Default value is \"CPU\"."; +static const char target_device_message_fd[] = "Optional. Target device for Face Detection network (the list of available devices is shown below). " + "Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device. Default value is \"CPU\"."; +static const char target_device_message_hp[] = "Optional. Target device for Head Pose Estimation network (the list of available devices is shown below). " + "Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device. Default value is \"CPU\"."; +static const char target_device_message_lm[] = "Optional. Target device for Facial Landmarks Estimation network " + "(the list of available devices is shown below). Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for device specified. Default value is \"CPU\"."; static const char camera_resolution_message[] = "Optional. Set camera resolution in format WxH."; - -/// @brief Message for performance counters static const char performance_counter_message[] = "Optional. Enable per-layer performance report."; - -/// @brief Message for probability threshold argument static const char thresh_output_message[] = "Optional. Probability threshold for Face Detector. The default value is 0.5."; - -/// @brief Message raw output flag static const char raw_output_message[] = "Optional. Output inference results as raw values."; - -/// @brief Message for enabling Face Detector network reshape static const char fd_reshape_message[] = "Optional. Reshape Face Detector network so that its input resolution has the same aspect ratio as the input frame."; - -/// @brief Message do not show processed video static const char no_show_processed_video[] = "Optional. Do not show processed video."; +static const char utilization_monitors_message[] = "Optional. List of monitors to show initially."; -/// \brief Define flag for showing help message
DEFINE_bool(h, false, help_message); - -/// \brief Define parameter for set image file
-/// It is a required parameter DEFINE_string(i, "cam", video_message); - -/// \brief Define parameter for Gaze Estimation model file
-/// It is a required parameter DEFINE_string(m, "", gaze_estimation_model_message); - -/// \brief Define parameter for Face Detection model file
-/// It is a required parameter DEFINE_string(m_fd, "", face_detection_model_message); - -/// \brief Define parameter for Head Pose Estimation model file
-/// It is a required parameter DEFINE_string(m_hp, "", head_pose_model_message); - -/// \brief Define parameter for Facial Landmarks Estimation model file
-/// It is an optional parameter DEFINE_string(m_lm, "", facial_landmarks_model_message); - -/// \brief target device for Gaze Estimation network
DEFINE_string(d, "CPU", target_device_message); - -/// \brief Define parameter for target device for Face Detection network
DEFINE_string(d_fd, "CPU", target_device_message_fd); - -/// \brief Define parameter for target device for Head Pose Estimation network
DEFINE_string(d_hp, "CPU", target_device_message_hp); - -/// \brief Define parameter for target device for Facial Landmarks Estimation network
DEFINE_string(d_lm, "CPU", target_device_message_lm); - -/// \brief Define parameter camera resolution
-/// It is an optional parameter DEFINE_string(res, "", camera_resolution_message); - -/// \brief Define parameter to enable face detector network reshape
-/// It is an optional parameter DEFINE_bool(fd_reshape, false, fd_reshape_message); - -/// \brief Define parameter to enable per-layer performance report
-/// It is an optional parameter DEFINE_bool(pc, false, performance_counter_message); - -/// \brief Define a flag to output raw scoring results
-/// It is an optional parameter DEFINE_bool(r, false, raw_output_message); - -/// \brief Define a parameter for probability threshold for detections
-/// It is an optional parameter DEFINE_double(t, 0.5, thresh_output_message); - -/// \brief Define a flag to disable showing processed video
-/// It is an optional parameter DEFINE_bool(no_show, false, no_show_processed_video); +DEFINE_string(u, "", utilization_monitors_message); /** * \brief This function shows a help message @@ -154,4 +87,5 @@ static void showUsage() { std::cout << " -pc " << performance_counter_message << std::endl; std::cout << " -r " << raw_output_message << std::endl; std::cout << " -t " << thresh_output_message << std::endl; + std::cout << " -u " << utilization_monitors_message << std::endl; } diff --git a/demos/gaze_estimation_demo/include/face_detector.hpp b/demos/gaze_estimation_demo/include/face_detector.hpp index 7c13b4d766f..53d13388b57 100644 --- a/demos/gaze_estimation_demo/include/face_detector.hpp +++ b/demos/gaze_estimation_demo/include/face_detector.hpp @@ -28,6 +28,11 @@ class FaceDetector { private: IEWrapper ieWrapper; + std::string inputBlobName; + std::vector inputBlobDims; + std::string outputBlobName; + std::size_t numTotalDetections; + double detectionThreshold; bool enableReshape; diff --git a/demos/gaze_estimation_demo/include/gaze_estimator.hpp b/demos/gaze_estimation_demo/include/gaze_estimator.hpp index e025fae96ca..143928152ea 100644 --- a/demos/gaze_estimation_demo/include/gaze_estimator.hpp +++ b/demos/gaze_estimation_demo/include/gaze_estimator.hpp @@ -26,6 +26,7 @@ class GazeEstimator: public BaseEstimator { private: IEWrapper ieWrapper; + std::string outputBlobName; bool rollAlign; cv::Rect createEyeBoundingBox(const cv::Point2i& p1, const cv::Point2i& p2, float scale = 1.8) const; void rotateImageAroundCenter(const cv::Mat& srcImage, cv::Mat& dstImage, float angle) const; diff --git a/demos/gaze_estimation_demo/include/head_pose_estimator.hpp b/demos/gaze_estimation_demo/include/head_pose_estimator.hpp index 1d5f2e4d212..96cc396fbaa 100644 --- a/demos/gaze_estimation_demo/include/head_pose_estimator.hpp +++ b/demos/gaze_estimation_demo/include/head_pose_estimator.hpp @@ -25,5 +25,6 @@ class HeadPoseEstimator: public BaseEstimator { private: IEWrapper ieWrapper; + std::string inputBlobName; }; } // namespace gaze_estimation diff --git a/demos/gaze_estimation_demo/include/ie_wrapper.hpp b/demos/gaze_estimation_demo/include/ie_wrapper.hpp index 6e755c898e2..ce5f43b4105 100644 --- a/demos/gaze_estimation_demo/include/ie_wrapper.hpp +++ b/demos/gaze_estimation_demo/include/ie_wrapper.hpp @@ -30,16 +30,19 @@ class IEWrapper { // For setting input blobs containing vectors of data void setInputBlob(const std::string& blobName, const std::vector& data); - // Get output blob content as a vector given its name (if there are more than one output blob) + // Get output blob content as a vector given its name void getOutputBlob(const std::string& blobName, std::vector& output); - // Get output blob content as a vector (if there is only one output blob) - void getOutputBlob(std::vector& output); void printPerlayerPerformance() const; - const std::map>& getIputBlobDimsInfo() const; + const std::map>& getInputBlobDimsInfo() const; const std::map>& getOutputBlobDimsInfo() const; + std::string expectSingleInput() const; + std::string expectSingleOutput() const; + + void expectImageInput(const std::string& blobName) const; + void reshape(const std::map>& newBlobsDimsInfo); void infer(); @@ -48,7 +51,6 @@ class IEWrapper { std::string modelPath; std::string deviceName; InferenceEngine::Core& ie; - InferenceEngine::CNNNetReader netReader; InferenceEngine::CNNNetwork network; InferenceEngine::ExecutableNetwork executableNetwork; InferenceEngine::InferRequest request; diff --git a/demos/gaze_estimation_demo/include/landmarks_estimator.hpp b/demos/gaze_estimation_demo/include/landmarks_estimator.hpp index ed834a8589b..f65ab817a2d 100644 --- a/demos/gaze_estimation_demo/include/landmarks_estimator.hpp +++ b/demos/gaze_estimation_demo/include/landmarks_estimator.hpp @@ -25,5 +25,6 @@ class LandmarksEstimator: public BaseEstimator { private: IEWrapper ieWrapper; + std::string inputBlobName, outputBlobName; }; } // namespace gaze_estimation diff --git a/demos/gaze_estimation_demo/main.cpp b/demos/gaze_estimation_demo/main.cpp index e3bb7cef3d6..e0df0025241 100644 --- a/demos/gaze_estimation_demo/main.cpp +++ b/demos/gaze_estimation_demo/main.cpp @@ -28,6 +28,7 @@ #include +#include #include #include @@ -49,9 +50,6 @@ #include "utils.hpp" #include -#ifdef WITH_EXTENSIONS -#include -#endif using namespace InferenceEngine; using namespace gaze_estimation; @@ -140,8 +138,8 @@ int main(int argc, char *argv[]) { LandmarksEstimator landmarksEstimator(ie, FLAGS_m_lm, FLAGS_d_lm); GazeEstimator gazeEstimator(ie, FLAGS_m, FLAGS_d); - // Put pointers to all estimators in a vector so that they could be processed uniformly in a loop - std::vector estimators = {&headPoseEstimator, &landmarksEstimator, &gazeEstimator}; + // Put pointers to all estimators in an array so that they could be processed uniformly in a loop + BaseEstimator* estimators[] = {&headPoseEstimator, &landmarksEstimator, &gazeEstimator}; // Each element of the vector contains inference results on one face std::vector inferenceResults; @@ -153,7 +151,8 @@ int main(int argc, char *argv[]) { int delay = 1; std::string windowName = "Gaze estimation demo"; - double overallTime = 0., inferenceTime = 0.; + cv::Size graphSize{static_cast(cap.get(cv::CAP_PROP_FRAME_WIDTH) / 4), 60}; + Presenter presenter(FLAGS_u, static_cast(cap.get(cv::CAP_PROP_FRAME_HEIGHT)) - graphSize.height - 10, graphSize); auto tIterationBegins = cv::getTickCount(); do { if (flipImage) { @@ -172,11 +171,11 @@ int main(int argc, char *argv[]) { // Measure FPS auto tIterationEnds = cv::getTickCount(); - overallTime = (tIterationEnds - tIterationBegins) * 1000. / cv::getTickFrequency(); + double overallTime = (tIterationEnds - tIterationBegins) * 1000. / cv::getTickFrequency(); overallTimeAverager.updateValue(overallTime); tIterationBegins = tIterationEnds; - inferenceTime = (tInferenceEnds - tInferenceBegins) * 1000. / cv::getTickFrequency(); + double inferenceTime = (tInferenceEnds - tInferenceBegins) * 1000. / cv::getTickFrequency(); inferenceTimeAverager.updateValue(inferenceTime); if (FLAGS_pc) { @@ -196,6 +195,8 @@ int main(int argc, char *argv[]) { continue; } + presenter.drawGraphs(frame); + // Display the results for (auto const& inferenceResult : inferenceResults) { resultsMarker.mark(frame, inferenceResult); @@ -213,7 +214,10 @@ int main(int argc, char *argv[]) { break; else if (key == 'f') flipImage = !flipImage; + else + presenter.handleKey(key); } while (cap.read(frame)); + std::cout << presenter.reportMeans() << '\n'; } catch (const std::exception& error) { slog::err << error.what() << slog::endl; diff --git a/demos/gaze_estimation_demo/src/face_detector.cpp b/demos/gaze_estimation_demo/src/face_detector.cpp index 75de6ad8fd7..8bc190ae0d7 100644 --- a/demos/gaze_estimation_demo/src/face_detector.cpp +++ b/demos/gaze_estimation_demo/src/face_detector.cpp @@ -19,6 +19,22 @@ FaceDetector::FaceDetector(InferenceEngine::Core& ie, ieWrapper(ie, modelPath, deviceName), detectionThreshold(detectionConfidenceThreshold), enableReshape(enableReshape) { + const auto& inputInfo = ieWrapper.getInputBlobDimsInfo(); + + inputBlobName = ieWrapper.expectSingleInput(); + ieWrapper.expectImageInput(inputBlobName); + inputBlobDims = inputInfo.at(inputBlobName); + + const auto& outputInfo = ieWrapper.getOutputBlobDimsInfo(); + + outputBlobName = ieWrapper.expectSingleOutput(); + const auto& outputBlobDims = outputInfo.at(outputBlobName); + + if (outputBlobDims.size() != 4 || outputBlobDims[0] != 1 || outputBlobDims[1] != 1 || outputBlobDims[3] != 7) { + throw std::runtime_error(modelPath + ": expected \"" + outputBlobName + "\" to have shape 1x1xNx7"); + } + + numTotalDetections = outputBlobDims[2]; } void FaceDetector::adjustBoundingBox(cv::Rect& boundingBox) const { @@ -45,10 +61,6 @@ void FaceDetector::adjustBoundingBox(cv::Rect& boundingBox) const { std::vector FaceDetector::detect(const cv::Mat& image) { std::vector detectionResult; - auto ieInputBlobInfo = ieWrapper.getIputBlobDimsInfo().begin(); - auto inputBlobName = ieInputBlobInfo->first; - auto inputBlobDims = ieInputBlobInfo->second; - if (enableReshape) { double imageAspectRatio = std::round(100. * image.cols / image.rows) / 100.; double networkAspectRatio = std::round(100. * inputBlobDims[3] / inputBlobDims[2]) / 100.; @@ -56,12 +68,11 @@ std::vector FaceDetector::detect(const cv::Mat& image) { if (std::fabs(imageAspectRatio - networkAspectRatio) > aspectRatioThreshold) { std::cout << "Face Detection network is reshaped" << std::endl; - std::map> newBlobsDimsInfo; - auto newBlobDims(inputBlobDims); + // Fix height and change width to make networkAspectRatio equal to imageAspectRatio - newBlobDims[3] = static_cast(newBlobDims[2] * imageAspectRatio); - newBlobsDimsInfo[inputBlobName] = newBlobDims; - ieWrapper.reshape(newBlobsDimsInfo); + inputBlobDims[3] = static_cast(inputBlobDims[2] * imageAspectRatio); + + ieWrapper.reshape({{inputBlobName, inputBlobDims}}); } } @@ -69,27 +80,23 @@ std::vector FaceDetector::detect(const cv::Mat& image) { ieWrapper.infer(); std::vector rawDetectionResults; - ieWrapper.getOutputBlob(rawDetectionResults); - auto outputBlobDims = ieWrapper.getOutputBlobDimsInfo().begin()->second; - - auto nTotalDetections = outputBlobDims[2]; - auto nInfoFields = outputBlobDims[3]; + ieWrapper.getOutputBlob(outputBlobName, rawDetectionResults); FaceInferenceResults tmp; cv::Size imageSize(image.size()); cv::Rect imageRect(0, 0, image.cols, image.rows); - for (unsigned long detectionID = 0; detectionID < nTotalDetections; ++detectionID) { - float confidence = rawDetectionResults[detectionID * nInfoFields + 2]; + for (unsigned long detectionID = 0; detectionID < numTotalDetections; ++detectionID) { + float confidence = rawDetectionResults[detectionID * 7 + 2]; if (static_cast(confidence) < detectionThreshold) { break; } - auto x = rawDetectionResults[detectionID * nInfoFields + 3] * imageSize.width; - auto width = rawDetectionResults[detectionID * nInfoFields + 5] * imageSize.width - x; - auto y = rawDetectionResults[detectionID * nInfoFields + 4] * imageSize.height; - auto height = rawDetectionResults[detectionID * nInfoFields + 6] * imageSize.height - y; + auto x = rawDetectionResults[detectionID * 7 + 3] * imageSize.width; + auto width = rawDetectionResults[detectionID * 7 + 5] * imageSize.width - x; + auto y = rawDetectionResults[detectionID * 7 + 4] * imageSize.height; + auto height = rawDetectionResults[detectionID * 7 + 6] * imageSize.height - y; cv::Rect faceRect(static_cast(x), static_cast(y), static_cast(width), static_cast(height)); diff --git a/demos/gaze_estimation_demo/src/gaze_estimator.cpp b/demos/gaze_estimation_demo/src/gaze_estimator.cpp index 3e12fc3ac78..d84828b27ce 100644 --- a/demos/gaze_estimation_demo/src/gaze_estimator.cpp +++ b/demos/gaze_estimation_demo/src/gaze_estimator.cpp @@ -9,11 +9,42 @@ #include "gaze_estimator.hpp" namespace gaze_estimation { + +const char BLOB_HEAD_POSE_ANGLES[] = "head_pose_angles"; +const char BLOB_LEFT_EYE_IMAGE[] = "left_eye_image"; +const char BLOB_RIGHT_EYE_IMAGE[] = "right_eye_image"; + GazeEstimator::GazeEstimator(InferenceEngine::Core& ie, const std::string& modelPath, const std::string& deviceName, bool doRollAlign): ieWrapper(ie, modelPath, deviceName), rollAlign(doRollAlign) { + const auto& inputInfo = ieWrapper.getInputBlobDimsInfo(); + + for (const auto& blobName: {BLOB_HEAD_POSE_ANGLES, BLOB_LEFT_EYE_IMAGE, BLOB_RIGHT_EYE_IMAGE}) { + if (inputInfo.find(blobName) == inputInfo.end()) + throw std::runtime_error(modelPath + ": expected to have input named \"" + blobName + "\""); + } + + auto expectAngles = [&modelPath](const std::string& blobName, const std::vector& dims) { + bool is1Dim = !dims.empty() + && std::all_of(dims.begin(), dims.end() - 1, [](unsigned long n) { return n == 1; }); + + if (!is1Dim || dims.back() != 3) { + throw std::runtime_error(modelPath + ": expected \"" + blobName + "\" to have dimensions [1x...]3"); + } + }; + + expectAngles(BLOB_HEAD_POSE_ANGLES, inputInfo.at(BLOB_HEAD_POSE_ANGLES)); + + for (const auto& blobName: {BLOB_LEFT_EYE_IMAGE, BLOB_RIGHT_EYE_IMAGE}) { + ieWrapper.expectImageInput(blobName); + } + + const auto& outputInfo = ieWrapper.getOutputBlobDimsInfo(); + + outputBlobName = ieWrapper.expectSingleOutput(); + expectAngles(outputBlobName, outputInfo.at(outputBlobName)); } cv::Rect GazeEstimator::createEyeBoundingBox(const cv::Point2i& p1, @@ -82,15 +113,15 @@ void GazeEstimator::estimate(const cv::Mat& image, rightEyeImage = rightEyeImageRotated; } - ieWrapper.setInputBlob("head_pose_angles", headPoseAngles); - ieWrapper.setInputBlob("left_eye_image", leftEyeImage); - ieWrapper.setInputBlob("right_eye_image", rightEyeImage); + ieWrapper.setInputBlob(BLOB_HEAD_POSE_ANGLES, headPoseAngles); + ieWrapper.setInputBlob(BLOB_LEFT_EYE_IMAGE, leftEyeImage); + ieWrapper.setInputBlob(BLOB_RIGHT_EYE_IMAGE, rightEyeImage); ieWrapper.infer(); std::vector rawResults; - ieWrapper.getOutputBlob(rawResults); + ieWrapper.getOutputBlob(outputBlobName, rawResults); cv::Point3f gazeVector; gazeVector.x = rawResults[0]; diff --git a/demos/gaze_estimation_demo/src/head_pose_estimator.cpp b/demos/gaze_estimation_demo/src/head_pose_estimator.cpp index 15543ac9a07..8c3ded1c64a 100644 --- a/demos/gaze_estimation_demo/src/head_pose_estimator.cpp +++ b/demos/gaze_estimation_demo/src/head_pose_estimator.cpp @@ -8,10 +8,35 @@ #include "head_pose_estimator.hpp" namespace gaze_estimation { + +const std::pair OUTPUTS[] = { + {"angle_y_fc", &cv::Point3f::x}, + {"angle_p_fc", &cv::Point3f::y}, + {"angle_r_fc", &cv::Point3f::z}, +}; + HeadPoseEstimator::HeadPoseEstimator(InferenceEngine::Core& ie, const std::string& modelPath, const std::string& deviceName): ieWrapper(ie, modelPath, deviceName) { + inputBlobName = ieWrapper.expectSingleInput(); + ieWrapper.expectImageInput(inputBlobName); + + const auto& outputInfo = ieWrapper.getOutputBlobDimsInfo(); + + for (const auto& output: OUTPUTS) { + auto it = outputInfo.find(output.first); + + if (it == outputInfo.end()) + throw std::runtime_error( + modelPath + ": expected to have output named \"" + output.first + "\""); + + bool correctDims = std::all_of(it->second.begin(), it->second.end(), + [](unsigned long n) { return n == 1; }); + if (!correctDims) + throw std::runtime_error( + modelPath + ": expected \"" + output.first + "\" to have total size 1"); + } } void HeadPoseEstimator::estimate(const cv::Mat& image, @@ -19,19 +44,15 @@ void HeadPoseEstimator::estimate(const cv::Mat& image, auto faceBoundingBox = outputResults.faceBoundingBox; auto faceCrop(cv::Mat(image, faceBoundingBox)); - auto inputBlobName = ieWrapper.getIputBlobDimsInfo().begin()->first; - ieWrapper.setInputBlob(inputBlobName, faceCrop); ieWrapper.infer(); - std::vector y, p, r; - ieWrapper.getOutputBlob("angle_y_fc", y); - ieWrapper.getOutputBlob("angle_p_fc", p); - ieWrapper.getOutputBlob("angle_r_fc", r); + std::vector outputValue; - outputResults.headPoseAngles.x = y[0]; - outputResults.headPoseAngles.y = p[0]; - outputResults.headPoseAngles.z = r[0]; + for (const auto &output: OUTPUTS) { + ieWrapper.getOutputBlob(output.first, outputValue); + outputResults.headPoseAngles.*output.second = outputValue[0]; + } } void HeadPoseEstimator::printPerformanceCounts() const { diff --git a/demos/gaze_estimation_demo/src/ie_wrapper.cpp b/demos/gaze_estimation_demo/src/ie_wrapper.cpp index 62455fcaf48..d33994cdecf 100644 --- a/demos/gaze_estimation_demo/src/ie_wrapper.cpp +++ b/demos/gaze_estimation_demo/src/ie_wrapper.cpp @@ -16,10 +16,7 @@ IEWrapper::IEWrapper(InferenceEngine::Core& ie, const std::string& modelPath, const std::string& deviceName): modelPath(modelPath), deviceName(deviceName), ie(ie) { - netReader.ReadNetwork(modelPath); - std::string binFileName = fileNameNoExt(modelPath) + ".bin"; - netReader.ReadWeights(binFileName); - network = netReader.getNetwork(); + network = ie.ReadNetwork(modelPath); setExecPart(); } @@ -110,27 +107,35 @@ void IEWrapper::getOutputBlob(const std::string& blobName, } } -void IEWrapper::getOutputBlob(std::vector& output) { - output.clear(); - auto blobName = outputBlobsDimsInfo.begin()->first; - auto blobDims = outputBlobsDimsInfo[blobName]; - auto dataSize = 1; - for (auto const& dim : blobDims) { - dataSize *= dim; - } - auto outputBlob = request.GetBlob(blobName); - auto buffer = outputBlob->buffer().as::value_type *>(); +const std::map>& IEWrapper::getInputBlobDimsInfo() const { + return inputBlobsDimsInfo; +} +const std::map>& IEWrapper::getOutputBlobDimsInfo() const { + return outputBlobsDimsInfo; +} - for (int i = 0; i < dataSize; ++i) { - output.push_back(buffer[i]); +std::string IEWrapper::expectSingleInput() const { + if (inputBlobsDimsInfo.size() != 1) { + throw std::runtime_error(modelPath + ": expected to have 1 input"); } + + return inputBlobsDimsInfo.begin()->first; } -const std::map>& IEWrapper::getIputBlobDimsInfo() const { - return inputBlobsDimsInfo; +std::string IEWrapper::expectSingleOutput() const { + if (outputBlobsDimsInfo.size() != 1) { + throw std::runtime_error(modelPath + ": expected to have 1 output"); + } + + return outputBlobsDimsInfo.begin()->first; } -const std::map>& IEWrapper::getOutputBlobDimsInfo() const { - return outputBlobsDimsInfo; + +void IEWrapper::expectImageInput(const std::string& blobName) const { + const auto& dims = inputBlobsDimsInfo.at(blobName); + + if (dims.size() != 4 || dims[0] != 1 || dims[1] != 3) { + throw std::runtime_error(modelPath + ": expected \"" + blobName + "\" to have dimensions 1x3xHxW"); + } } void IEWrapper::infer() { diff --git a/demos/gaze_estimation_demo/src/landmarks_estimator.cpp b/demos/gaze_estimation_demo/src/landmarks_estimator.cpp index ee9a12e4d21..8e994916b17 100644 --- a/demos/gaze_estimation_demo/src/landmarks_estimator.cpp +++ b/demos/gaze_estimation_demo/src/landmarks_estimator.cpp @@ -12,6 +12,21 @@ LandmarksEstimator::LandmarksEstimator(InferenceEngine::Core& ie, const std::string& modelPath, const std::string& deviceName): ieWrapper(ie, modelPath, deviceName) { + inputBlobName = ieWrapper.expectSingleInput(); + ieWrapper.expectImageInput(inputBlobName); + + const auto& outputInfo = ieWrapper.getOutputBlobDimsInfo(); + + outputBlobName = ieWrapper.expectSingleOutput(); + const auto& outputBlobDims = outputInfo.at(outputBlobName); + + bool outputIs1Dim = !outputBlobDims.empty() + && std::all_of(outputBlobDims.begin(), outputBlobDims.end() - 1, + [](unsigned long n) { return n == 1; }); + + if (!outputIs1Dim || outputBlobDims.back() % 2 != 0) { + throw std::runtime_error(modelPath + ": expected \"" + outputBlobName + "\" to have dimensions [1x...]2N"); + } } void LandmarksEstimator::estimate(const cv::Mat& image, @@ -19,13 +34,11 @@ void LandmarksEstimator::estimate(const cv::Mat& image, auto faceBoundingBox = outputResults.faceBoundingBox; auto faceCrop(cv::Mat(image, faceBoundingBox)); - auto inputBlobName = ieWrapper.getIputBlobDimsInfo().begin()->first; - ieWrapper.setInputBlob(inputBlobName, faceCrop); ieWrapper.infer(); std::vector rawLandmarks; - ieWrapper.getOutputBlob(rawLandmarks); + ieWrapper.getOutputBlob(outputBlobName, rawLandmarks); for (unsigned long i = 0; i < rawLandmarks.size() / 2; ++i) { int x = static_cast(rawLandmarks[2 * i] * faceCrop.cols + faceBoundingBox.tl().x); diff --git a/demos/gaze_estimation_demo/src/utils.cpp b/demos/gaze_estimation_demo/src/utils.cpp index b8349228082..a66897f00d0 100644 --- a/demos/gaze_estimation_demo/src/utils.cpp +++ b/demos/gaze_estimation_demo/src/utils.cpp @@ -38,9 +38,6 @@ void initializeIEObject(InferenceEngine::Core& ie, /** Loading extensions for the CPU device **/ if ((deviceName.find("CPU") != std::string::npos)) { -#ifdef WITH_EXTENSIONS - ie.AddExtension(std::make_shared(), "CPU"); -#endif loadedDevices.insert(deviceName); } } diff --git a/demos/human_pose_estimation_demo/CMakeLists.txt b/demos/human_pose_estimation_demo/CMakeLists.txt index f50c924ce38..ddce9118dce 100644 --- a/demos/human_pose_estimation_demo/CMakeLists.txt +++ b/demos/human_pose_estimation_demo/CMakeLists.txt @@ -9,4 +9,5 @@ ie_add_sample(NAME human_pose_estimation_demo SOURCES ${SOURCES} HEADERS ${HEADERS} INCLUDE_DIRECTORIES "${CMAKE_CURRENT_SOURCE_DIR}/include" + DEPENDENCIES monitors OPENCV_DEPENDENCIES highgui) diff --git a/demos/human_pose_estimation_demo/README.md b/demos/human_pose_estimation_demo/README.md index ce01fe8d310..65d555b58f2 100644 --- a/demos/human_pose_estimation_demo/README.md +++ b/demos/human_pose_estimation_demo/README.md @@ -21,7 +21,7 @@ On the start-up, the application reads command line parameters and loads human p ## Running Running the application with the `-h` option yields the following usage message: -```sh +``` ./human_pose_estimation_demo -h InferenceEngine: API version ............ @@ -36,8 +36,9 @@ Options: -d "" Optional. Specify the target device for Human Pose Estimation (the list of available devices is shown below). Default value is CPU. Use "-d HETERO:" format to specify HETERO plugin. The application looks for a suitable plugin for the specified device. -pc Optional. Enable per-layer performance report. -no_show Optional. Do not show processed video. + -black Optional. Show black background. -r Optional. Output inference results as raw values. - + -u Optional. List of monitors to show initially. ``` Running the application with an empty list of options yields an error message. diff --git a/demos/human_pose_estimation_demo/include/human_pose_estimation_demo.hpp b/demos/human_pose_estimation_demo/include/human_pose_estimation_demo.hpp index a410aa24eaa..df46811490c 100644 --- a/demos/human_pose_estimation_demo/include/human_pose_estimation_demo.hpp +++ b/demos/human_pose_estimation_demo/include/human_pose_estimation_demo.hpp @@ -7,56 +7,28 @@ #include #include -/// @brief Message for help argument static const char help_message[] = "Print a usage message."; - -/// @brief Message for video argument static const char video_message[] = "Required. Path to a video. Default value is \"cam\" to work with camera."; - -/// @brief Message for model argument static const char human_pose_estimation_model_message[] = "Required. Path to the Human Pose Estimation model (.xml) file."; - -/// @brief Message for assigning Human Pose Estimation inference to device -static const char target_device_message[] = "Optional. Specify the target device for Human Pose Estimation "\ - "(the list of available devices is shown below). Default value is CPU. " \ - "Use \"-d HETERO:\" format to specify HETERO plugin. " \ +static const char target_device_message[] = "Optional. Specify the target device for Human Pose Estimation " + "(the list of available devices is shown below). Default value is CPU. " + "Use \"-d HETERO:\" format to specify HETERO plugin. " "The application looks for a suitable plugin for the specified device."; - -/// @brief Message for performance counter static const char performance_counter_message[] = "Optional. Enable per-layer performance report."; - -/// @brief Message for not showing processed video static const char no_show_processed_video[] = "Optional. Do not show processed video."; - -/// @brief Message for raw output +static const char black_background[] = "Optional. Show black background."; static const char raw_output_message[] = "Optional. Output inference results as raw values."; +static const char utilization_monitors_message[] = "Optional. List of monitors to show initially."; -/// @brief Defines flag for showing help message
DEFINE_bool(h, false, help_message); - -/// @brief Defines parameter for setting video file
-/// It is a required parameter DEFINE_string(i, "cam", video_message); - -/// @brief Defines parameter for human pose estimation model file
-/// It is a required parameter DEFINE_string(m, "", human_pose_estimation_model_message); - -/// @brief Defines parameter for the target device to infer on
-/// It is an optional parameter DEFINE_string(d, "CPU", target_device_message); - -/// @brief Defines flag for per-layer performance report
-/// It is an optional parameter DEFINE_bool(pc, false, performance_counter_message); - -/// @brief Defines flag for disabling processed video showing
-/// It is an optional parameter DEFINE_bool(no_show, false, no_show_processed_video); - -/// @brief Defines flag to output raw results
-/// It is an optional parameter +DEFINE_bool(black, false, black_background); DEFINE_bool(r, false, raw_output_message); +DEFINE_string(u, "", utilization_monitors_message); /** * @brief This function shows a help message @@ -72,5 +44,7 @@ static void showUsage() { std::cout << " -d \"\" " << target_device_message << std::endl; std::cout << " -pc " << performance_counter_message << std::endl; std::cout << " -no_show " << no_show_processed_video << std::endl; + std::cout << " -black " << black_background << std::endl; std::cout << " -r " << raw_output_message << std::endl; + std::cout << " -u " << utilization_monitors_message << std::endl; } diff --git a/demos/human_pose_estimation_demo/include/human_pose_estimator.hpp b/demos/human_pose_estimation_demo/include/human_pose_estimator.hpp index b3b419d4757..c48b9a4391d 100644 --- a/demos/human_pose_estimation_demo/include/human_pose_estimator.hpp +++ b/demos/human_pose_estimation_demo/include/human_pose_estimator.hpp @@ -15,12 +15,19 @@ namespace human_pose_estimation { class HumanPoseEstimator { public: - static const size_t keypointsNumber; + static const size_t keypointsNumber = 18; HumanPoseEstimator(const std::string& modelPath, const std::string& targetDeviceName, bool enablePerformanceReport = false); - std::vector estimate(const cv::Mat& image); + std::vector postprocessCurr(); + void reshape(const cv::Mat& image); + void frameToBlobCurr(const cv::Mat& image); + void frameToBlobNext(const cv::Mat& image); + void startCurr(); + void startNext(); + bool readyCurr(); + void swapRequest(); ~HumanPoseEstimator(); private: @@ -47,13 +54,14 @@ class HumanPoseEstimator { float foundMidPointsRatioThreshold; float minSubsetScore; cv::Size inputLayerSize; + cv::Size imageSize; int upsampleRatio; InferenceEngine::Core ie; std::string targetDeviceName; InferenceEngine::CNNNetwork network; InferenceEngine::ExecutableNetwork executableNetwork; - InferenceEngine::InferRequest request; - InferenceEngine::CNNNetReader netReader; + InferenceEngine::InferRequest::Ptr requestNext; + InferenceEngine::InferRequest::Ptr requestCurr; std::string pafsBlobName; std::string heatmapsBlobName; bool enablePerformanceReport; diff --git a/demos/human_pose_estimation_demo/main.cpp b/demos/human_pose_estimation_demo/main.cpp index 224803d37e7..d73e3301369 100644 --- a/demos/human_pose_estimation_demo/main.cpp +++ b/demos/human_pose_estimation_demo/main.cpp @@ -9,9 +9,11 @@ */ #include +#include #include +#include #include #include "human_pose_estimation_demo.hpp" @@ -60,12 +62,15 @@ int main(int argc, char* argv[]) { } int delay = 33; - double inferenceTime = 0.0; - cv::Mat image; - if (!cap.read(image)) { + + // read input (video) frame + cv::Mat curr_frame; cap >> curr_frame; + cv::Mat next_frame; + if (!cap.grab()) { throw std::logic_error("Failed to get frame from cv::VideoCapture"); } - estimator.estimate(image); // Do not measure network reshape, if it happened + + estimator.reshape(curr_frame); // Do not measure network reshape, if it happened std::cout << "To close the application, press 'CTRL+C' here"; if (!FLAGS_no_show) { @@ -74,50 +79,159 @@ int main(int argc, char* argv[]) { } std::cout << std::endl; - do { - double t1 = static_cast(cv::getTickCount()); - std::vector poses = estimator.estimate(image); - double t2 = static_cast(cv::getTickCount()); - if (inferenceTime == 0) { - inferenceTime = (t2 - t1) / cv::getTickFrequency() * 1000; - } else { - inferenceTime = inferenceTime * 0.95 + 0.05 * (t2 - t1) / cv::getTickFrequency() * 1000; + cv::Size graphSize{static_cast(cap.get(cv::CAP_PROP_FRAME_WIDTH) / 4), 60}; + Presenter presenter(FLAGS_u, static_cast(cap.get(cv::CAP_PROP_FRAME_HEIGHT)) - graphSize.height - 10, graphSize); + std::vector poses; + bool isLastFrame = false; + bool isAsyncMode = false; // execution is always started in SYNC mode + bool isModeChanged = false; // set to true when execution mode is changed (SYNC<->ASYNC) + bool blackBackground = FLAGS_black; + + typedef std::chrono::duration> ms; + auto total_t0 = std::chrono::high_resolution_clock::now(); + auto wallclock = std::chrono::high_resolution_clock::now(); + double render_time = 0; + + while (true) { + auto t0 = std::chrono::high_resolution_clock::now(); + //here is the first asynchronus point: + //in the async mode we capture frame to populate the NEXT infer request + //in the regular mode we capture frame to the current infer request + + if (!cap.read(next_frame)) { + if (next_frame.empty()) { + isLastFrame = true; //end of video file + } else { + throw std::logic_error("Failed to get frame from cv::VideoCapture"); + } + } + if (isAsyncMode) { + if (isModeChanged) { + estimator.frameToBlobCurr(curr_frame); + } + if (!isLastFrame) { + estimator.frameToBlobNext(next_frame); + } + } else if (!isModeChanged) { + estimator.frameToBlobCurr(curr_frame); } - if (FLAGS_r) { - for (HumanPose const& pose : poses) { - std::stringstream rawPose; - rawPose << std::fixed << std::setprecision(0); - for (auto const& keypoint : pose.keypoints) { - rawPose << keypoint.x << "," << keypoint.y << " "; + auto t1 = std::chrono::high_resolution_clock::now(); + double decode_time = std::chrono::duration_cast(t1 - t0).count(); + + t0 = std::chrono::high_resolution_clock::now(); + // Main sync point: + // in the trully Async mode we start the NEXT infer request, while waiting for the CURRENT to complete + // in the regular mode we start the CURRENT request and immediately wait for it's completion + if (isAsyncMode) { + if (isModeChanged) { + estimator.startCurr(); + } + if (!isLastFrame) { + estimator.startNext(); + } + } else if (!isModeChanged) { + estimator.startCurr(); + } + + if (estimator.readyCurr()) { + t1 = std::chrono::high_resolution_clock::now(); + ms detection = std::chrono::duration_cast(t1 - t0); + t0 = std::chrono::high_resolution_clock::now(); + ms wall = std::chrono::duration_cast(t0 - wallclock); + wallclock = t0; + + t0 = std::chrono::high_resolution_clock::now(); + + if (!FLAGS_no_show) { + if (blackBackground) { + curr_frame = cv::Mat::zeros(curr_frame.size(), curr_frame.type()); } - rawPose << pose.score; - std::cout << rawPose.str() << std::endl; + std::ostringstream out; + out << "OpenCV cap/render time: " << std::fixed << std::setprecision(2) + << (decode_time + render_time) << " ms"; + + cv::putText(curr_frame, out.str(), cv::Point2f(0, 25), + cv::FONT_HERSHEY_TRIPLEX, 0.6, cv::Scalar(0, 255, 0)); + out.str(""); + out << "Wallclock time " << (isAsyncMode ? "(TRUE ASYNC): " : "(SYNC, press Tab): "); + out << std::fixed << std::setprecision(2) << wall.count() + << " ms (" << 1000.f / wall.count() << " fps)"; + cv::putText(curr_frame, out.str(), cv::Point2f(0, 50), + cv::FONT_HERSHEY_TRIPLEX, 0.6, cv::Scalar(0, 0, 255)); + if (!isAsyncMode) { // In the true async mode, there is no way to measure detection time directly + out.str(""); + out << "Detection time : " << std::fixed << std::setprecision(2) << detection.count() + << " ms (" + << 1000.f / detection.count() << " fps)"; + cv::putText(curr_frame, out.str(), cv::Point2f(0, 75), cv::FONT_HERSHEY_TRIPLEX, 0.6, + cv::Scalar(255, 0, 0)); + } + } + + poses = estimator.postprocessCurr(); + + if (FLAGS_r) { + if (!poses.empty()) { + std::time_t result = std::time(nullptr); + char timeString[sizeof("2020-01-01 00:00:00: ")]; + std::strftime(timeString, sizeof(timeString), "%Y-%m-%d %H:%M:%S: ", std::localtime(&result)); + std::cout << timeString; + } + + for (HumanPose const& pose : poses) { + std::stringstream rawPose; + rawPose << std::fixed << std::setprecision(0); + for (auto const& keypoint : pose.keypoints) { + rawPose << keypoint.x << "," << keypoint.y << " "; + } + rawPose << pose.score; + std::cout << rawPose.str() << std::endl; + } + } + + if (!FLAGS_no_show) { + presenter.drawGraphs(curr_frame); + renderHumanPose(poses, curr_frame); + cv::imshow("Human Pose Estimation on " + FLAGS_d, curr_frame); + t1 = std::chrono::high_resolution_clock::now(); + render_time = std::chrono::duration_cast(t1 - t0).count(); } } - if (FLAGS_no_show) { - continue; + if (isLastFrame) { + break; } - renderHumanPose(poses, image); + if (isModeChanged) { + isModeChanged = false; + } - cv::Mat fpsPane(35, 155, CV_8UC3); - fpsPane.setTo(cv::Scalar(153, 119, 76)); - cv::Mat srcRegion = image(cv::Rect(8, 8, fpsPane.cols, fpsPane.rows)); - cv::addWeighted(srcRegion, 0.4, fpsPane, 0.6, 0, srcRegion); - std::stringstream fpsSs; - fpsSs << "FPS: " << int(1000.0f / inferenceTime * 100) / 100.0f; - cv::putText(image, fpsSs.str(), cv::Point(16, 32), - cv::FONT_HERSHEY_COMPLEX, 0.8, cv::Scalar(0, 0, 255)); - cv::imshow("ICV Human Pose Estimation", image); + // Final point: + // in the truly Async mode we swap the NEXT and CURRENT requests for the next iteration + curr_frame = next_frame; + next_frame = cv::Mat(); + if (isAsyncMode) { + estimator.swapRequest(); + } - int key = cv::waitKey(delay) & 255; + const int key = cv::waitKey(delay) & 255; if (key == 'p') { delay = (delay == 0) ? 33 : 0; - } else if (key == 27) { + } else if (27 == key) { // Esc break; + } else if (9 == key) { // Tab + isAsyncMode ^= true; + isModeChanged = true; + } else if (32 == key) { // Space + blackBackground ^= true; } - } while (cap.read(image)); + presenter.handleKey(key); + } + + auto total_t1 = std::chrono::high_resolution_clock::now(); + ms total = std::chrono::duration_cast(total_t1 - total_t0); + std::cout << "Total Inference time: " << total.count() << std::endl; + std::cout << presenter.reportMeans() << '\n'; } catch (const std::exception& error) { std::cerr << "[ ERROR ] " << error.what() << std::endl; diff --git a/demos/human_pose_estimation_demo/src/human_pose_estimator.cpp b/demos/human_pose_estimation_demo/src/human_pose_estimator.cpp index 9b3a925177b..5b2fa35c370 100644 --- a/demos/human_pose_estimation_demo/src/human_pose_estimator.cpp +++ b/demos/human_pose_estimation_demo/src/human_pose_estimator.cpp @@ -14,8 +14,6 @@ #include "peak.hpp" namespace human_pose_estimation { -const size_t HumanPoseEstimator::keypointsNumber = 18; - HumanPoseEstimator::HumanPoseEstimator(const std::string& modelPath, const std::string& targetDeviceName_, bool enablePerformanceReport) @@ -36,27 +34,74 @@ HumanPoseEstimator::HumanPoseEstimator(const std::string& modelPath, ie.SetConfig({{InferenceEngine::PluginConfigParams::KEY_PERF_COUNT, InferenceEngine::PluginConfigParams::YES}}); } - netReader.ReadNetwork(modelPath); - std::string binFileName = fileNameNoExt(modelPath) + ".bin"; - netReader.ReadWeights(binFileName); - network = netReader.getNetwork(); - InferenceEngine::InputInfo::Ptr inputInfo = network.getInputsInfo().begin()->second; - inputLayerSize = cv::Size(inputInfo->getTensorDesc().getDims()[3], inputInfo->getTensorDesc().getDims()[2]); - inputInfo->setPrecision(InferenceEngine::Precision::U8); + network = ie.ReadNetwork(modelPath); + + const auto& inputInfo = network.getInputsInfo(); + + if (inputInfo.size() != 1) { + throw std::runtime_error(modelPath + ": expected to have 1 input"); + } + + const auto& imageInputInfo = *inputInfo.begin(); + const auto& imageInputDims = imageInputInfo.second->getTensorDesc().getDims(); + + if (imageInputDims.size() != 4 || imageInputDims[0] != 1 || imageInputDims[1] != 3) { + throw std::runtime_error( + modelPath + ": expected \"" + imageInputInfo.first + "\" to have dimensions 1x3xHxW"); + } + + inputLayerSize = cv::Size(imageInputDims[3], imageInputDims[2]); + imageInputInfo.second->setPrecision(InferenceEngine::Precision::U8); InferenceEngine::OutputsDataMap outputInfo = network.getOutputsInfo(); - auto outputBlobsIt = outputInfo.begin(); - pafsBlobName = outputBlobsIt->first; - heatmapsBlobName = (++outputBlobsIt)->first; + + if (outputInfo.size() != 2) { + throw std::runtime_error(modelPath + ": expected to have 2 outputs"); + } + + auto outputIt = outputInfo.begin(); + + const auto& pafsOutputInfo = *outputIt++; + + pafsBlobName = pafsOutputInfo.first; + + const auto& pafsOutputDims = pafsOutputInfo.second->getTensorDesc().getDims(); + + if (pafsOutputDims.size() != 4 || pafsOutputDims[0] != 1 + || pafsOutputDims[1] != 2 * (keypointsNumber + 1)) { + throw std::runtime_error( + modelPath + ": expected \"" + pafsBlobName + "\" to have dimensions " + "1x" + std::to_string(2 * (keypointsNumber + 1)) + "xHFMxWFM"); + } + + const auto& heatmapsOutputInfo = *outputIt++; + + heatmapsBlobName = heatmapsOutputInfo.first; + + const auto& heatmapsOutputDims = heatmapsOutputInfo.second->getTensorDesc().getDims(); + + if (heatmapsOutputDims.size() != 4 || heatmapsOutputDims[0] != 1 + || heatmapsOutputDims[1] != keypointsNumber + 1) { + throw std::runtime_error( + modelPath + ": expected \"" + heatmapsBlobName + "\" to have dimensions " + "1x" + std::to_string(keypointsNumber + 1) + "xHFMxWFM"); + } + + if (pafsOutputDims[2] != heatmapsOutputDims[2] || pafsOutputDims[3] != heatmapsOutputDims[3]) { + throw std::runtime_error( + modelPath + ": expected \"" + pafsBlobName + "\" and \"" + heatmapsBlobName + "\"" + "to have matching last two dimensions"); + } executableNetwork = ie.LoadNetwork(network, targetDeviceName); - request = executableNetwork.CreateInferRequest(); + requestNext = executableNetwork.CreateInferRequestPtr(); + requestCurr = executableNetwork.CreateInferRequestPtr(); } -std::vector HumanPoseEstimator::estimate(const cv::Mat& image) { +void HumanPoseEstimator::reshape(const cv::Mat& image){ CV_Assert(image.type() == CV_8UC3); - cv::Size imageSize = image.size(); + imageSize = image.size(); if (inputWidthIsChanged(imageSize)) { auto input_shapes = network.getInputShapes(); std::string input_name; @@ -67,19 +112,50 @@ std::vector HumanPoseEstimator::estimate(const cv::Mat& image) { input_shapes[input_name] = input_shape; network.reshape(input_shapes); executableNetwork = ie.LoadNetwork(network, targetDeviceName); - request = executableNetwork.CreateInferRequest(); + requestNext = executableNetwork.CreateInferRequestPtr(); + requestCurr = executableNetwork.CreateInferRequestPtr(); + std::cout << "Reshape needed" << std::endl; } - InferenceEngine::Blob::Ptr input = request.GetBlob(network.getInputsInfo().begin()->first); +} + +void HumanPoseEstimator::frameToBlobCurr(const cv::Mat& image) { + CV_Assert(image.type() == CV_8UC3); + InferenceEngine::Blob::Ptr input = requestCurr->GetBlob(network.getInputsInfo().begin()->first); + auto buffer = input->buffer().as::value_type *>(); + preprocess(image, buffer); +} + +void HumanPoseEstimator::frameToBlobNext(const cv::Mat& image) { + CV_Assert(image.type() == CV_8UC3); + InferenceEngine::Blob::Ptr input = requestNext->GetBlob(network.getInputsInfo().begin()->first); auto buffer = input->buffer().as::value_type *>(); preprocess(image, buffer); +} + +void HumanPoseEstimator::startCurr() { + requestCurr->StartAsync(); +} + +void HumanPoseEstimator::startNext() { + requestNext->StartAsync(); +} - request.Infer(); +bool HumanPoseEstimator::readyCurr() { + if (InferenceEngine::OK == requestCurr->Wait(InferenceEngine::IInferRequest::WaitMode::RESULT_READY)) { + return true; + } else { + return false; + } +} + +void HumanPoseEstimator::swapRequest() { + requestCurr.swap(requestNext); +} - InferenceEngine::Blob::Ptr pafsBlob = request.GetBlob(pafsBlobName); - InferenceEngine::Blob::Ptr heatMapsBlob = request.GetBlob(heatmapsBlobName); - CV_Assert(heatMapsBlob->getTensorDesc().getDims()[1] == keypointsNumber + 1); - InferenceEngine::SizeVector heatMapDims = - heatMapsBlob->getTensorDesc().getDims(); +std::vector HumanPoseEstimator::postprocessCurr() { + InferenceEngine::Blob::Ptr pafsBlob = requestCurr->GetBlob(pafsBlobName); + InferenceEngine::Blob::Ptr heatMapsBlob = requestCurr->GetBlob(heatmapsBlobName); + InferenceEngine::SizeVector heatMapDims = heatMapsBlob->getTensorDesc().getDims(); std::vector poses = postprocess( heatMapsBlob->buffer(), heatMapDims[2] * heatMapDims[3], @@ -101,8 +177,7 @@ void HumanPoseEstimator::preprocess(const cv::Mat& image, uint8_t* buffer) const cv::BORDER_CONSTANT, meanPixel); std::vector planes(3); for (size_t pId = 0; pId < planes.size(); pId++) { - planes[pId] = cv::Mat(inputLayerSize, CV_8UC1, - buffer + pId * inputLayerSize.area()); + planes[pId] = cv::Mat(inputLayerSize, CV_8UC1, buffer + pId * inputLayerSize.area()); } cv::split(paddedImage, planes); } @@ -233,7 +308,7 @@ HumanPoseEstimator::~HumanPoseEstimator() { try { if (enablePerformanceReport) { std::cout << "Performance counts for " << modelPath << std::endl << std::endl; - printPerformanceCounts(request, std::cout, getFullDeviceName(ie, targetDeviceName), false); + printPerformanceCounts(*requestCurr, std::cout, getFullDeviceName(ie, targetDeviceName), false); } } catch (...) { diff --git a/demos/human_pose_estimation_demo/src/peak.cpp b/demos/human_pose_estimation_demo/src/peak.cpp index a9bb1cb9eb7..26602293ff2 100644 --- a/demos/human_pose_estimation_demo/src/peak.cpp +++ b/demos/human_pose_estimation_demo/src/peak.cpp @@ -6,6 +6,8 @@ #include #include +#include + #include "peak.hpp" namespace human_pose_estimation { @@ -112,11 +114,11 @@ std::vector groupPeaksToPoses(const std::vector >& const float foundMidPointsRatioThreshold, const int minJointsNumber, const float minSubsetScore) { - const std::vector > limbIdsHeatmap = { + static const std::pair limbIdsHeatmap[] = { {2, 3}, {2, 6}, {3, 4}, {4, 5}, {6, 7}, {7, 8}, {2, 9}, {9, 10}, {10, 11}, {2, 12}, {12, 13}, {13, 14}, {2, 1}, {1, 15}, {15, 17}, {1, 16}, {16, 18}, {3, 17}, {6, 18} }; - const std::vector > limbIdsPaf = { + static const std::pair limbIdsPaf[] = { {31, 32}, {39, 40}, {33, 34}, {35, 36}, {41, 42}, {43, 44}, {19, 20}, {21, 22}, {23, 24}, {25, 26}, {27, 28}, {29, 30}, {47, 48}, {49, 50}, {53, 54}, {51, 52}, {55, 56}, {37, 38}, {45, 46} }; @@ -126,7 +128,7 @@ std::vector groupPeaksToPoses(const std::vector >& candidates.insert(candidates.end(), peaks.begin(), peaks.end()); } std::vector subset(0, HumanPoseByPeaksIndices(keypointsNumber)); - for (size_t k = 0; k < limbIdsPaf.size(); k++) { + for (size_t k = 0; k < arraySize(limbIdsPaf); k++) { std::vector connections; const int mapIdxOffset = keypointsNumber + 1; std::pair scoreMid = { pafs[limbIdsPaf[k].first - mapIdxOffset], diff --git a/demos/human_pose_estimation_demo/src/render_human_pose.cpp b/demos/human_pose_estimation_demo/src/render_human_pose.cpp index a2a45a479ce..7e29222a0e2 100644 --- a/demos/human_pose_estimation_demo/src/render_human_pose.cpp +++ b/demos/human_pose_estimation_demo/src/render_human_pose.cpp @@ -14,7 +14,7 @@ namespace human_pose_estimation { void renderHumanPose(const std::vector& poses, cv::Mat& image) { CV_Assert(image.type() == CV_8UC3); - const std::vector colors = { + static const cv::Scalar colors[HumanPoseEstimator::keypointsNumber] = { cv::Scalar(255, 0, 0), cv::Scalar(255, 85, 0), cv::Scalar(255, 170, 0), cv::Scalar(255, 255, 0), cv::Scalar(170, 255, 0), cv::Scalar(85, 255, 0), cv::Scalar(0, 255, 0), cv::Scalar(0, 255, 85), cv::Scalar(0, 255, 170), @@ -22,7 +22,7 @@ void renderHumanPose(const std::vector& poses, cv::Mat& image) { cv::Scalar(0, 0, 255), cv::Scalar(85, 0, 255), cv::Scalar(170, 0, 255), cv::Scalar(255, 0, 255), cv::Scalar(255, 0, 170), cv::Scalar(255, 0, 85) }; - const std::vector > limbKeypointsIds = { + static const std::pair limbKeypointsIds[] = { {1, 2}, {1, 5}, {2, 3}, {3, 4}, {5, 6}, {6, 7}, {1, 8}, {8, 9}, {9, 10}, diff --git a/demos/interactive_face_detection_demo/CMakeLists.txt b/demos/interactive_face_detection_demo/CMakeLists.txt index 55fc53c60cd..6dc75b6e9e7 100644 --- a/demos/interactive_face_detection_demo/CMakeLists.txt +++ b/demos/interactive_face_detection_demo/CMakeLists.txt @@ -8,4 +8,5 @@ file (GLOB MAIN_HEADERS ${CMAKE_CURRENT_SOURCE_DIR}/*.h*) ie_add_sample(NAME interactive_face_detection_demo SOURCES ${MAIN_SRC} HEADERS ${MAIN_HEADERS} + DEPENDENCIES monitors OPENCV_DEPENDENCIES highgui) diff --git a/demos/interactive_face_detection_demo/README.md b/demos/interactive_face_detection_demo/README.md index 055be3a04f6..a9fbb4a97c8 100644 --- a/demos/interactive_face_detection_demo/README.md +++ b/demos/interactive_face_detection_demo/README.md @@ -37,7 +37,7 @@ The new Async API operates with a new notion of the Infer Request that encapsula Running the application with the `-h` option yields the following usage message: -```sh +``` ./interactive_face_detection_demo -h InferenceEngine: API version ............ @@ -83,6 +83,7 @@ Options: -loop_video Optional. Enable playing video on a loop -no_smooth Optional. Do not smooth person attributes -no_show_emotion_bar Optional. Do not show emotion bar + -u Optional. List of monitors to show initially. ``` Running the application with an empty list of options yields the usage message given above and an error message. diff --git a/demos/interactive_face_detection_demo/detectors.cpp b/demos/interactive_face_detection_demo/detectors.cpp index b2135483c6e..e26bd3d6c6b 100644 --- a/demos/interactive_face_detection_demo/detectors.cpp +++ b/demos/interactive_face_detection_demo/detectors.cpp @@ -22,15 +22,12 @@ #include #include -#ifdef WITH_EXTENSIONS -#include -#endif #include "detectors.hpp" using namespace InferenceEngine; -BaseDetection::BaseDetection(std::string topoName, +BaseDetection::BaseDetection(const std::string &topoName, const std::string &pathToModel, const std::string &deviceForInference, int maxBatch, bool isBatchDynamic, bool isAsync, @@ -120,17 +117,13 @@ void FaceDetection::enqueue(const cv::Mat &frame) { enquedFrames = 1; } -CNNNetwork FaceDetection::read() { +CNNNetwork FaceDetection::read(const InferenceEngine::Core& ie) { slog::info << "Loading network files for Face Detection" << slog::endl; - CNNNetReader netReader; /** Read network model **/ - netReader.ReadNetwork(pathToModel); + auto network = ie.ReadNetwork(pathToModel); /** Set batch size to 1 **/ slog::info << "Batch size is set to " << maxBatch << slog::endl; - netReader.getNetwork().setBatchSize(maxBatch); - /** Extract model name and load its weights **/ - std::string binFileName = fileNameNoExt(pathToModel) + ".bin"; - netReader.ReadWeights(binFileName); + network.setBatchSize(maxBatch); /** Read labels (if any)**/ std::string labelFileName = fileNameNoExt(pathToModel) + ".labels"; @@ -143,7 +136,7 @@ CNNNetwork FaceDetection::read() { /** SSD-based network should have one input and one output **/ // ---------------------------Check inputs ------------------------------------------------------------- slog::info << "Checking Face Detection network inputs" << slog::endl; - InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Face Detection network should have only one input"); } @@ -153,14 +146,14 @@ CNNNetwork FaceDetection::read() { // ---------------------------Check outputs ------------------------------------------------------------ slog::info << "Checking Face Detection network outputs" << slog::endl; - OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 1) { throw std::logic_error("Face Detection network should have only one output"); } DataPtr& _output = outputInfo.begin()->second; output = outputInfo.begin()->first; - const CNNLayerPtr outputLayer = netReader.getNetwork().getLayerByName(output.c_str()); + const CNNLayerPtr outputLayer = network.getLayerByName(output.c_str()); if (outputLayer->type != "DetectionOutput") { throw std::logic_error("Face Detection network output layer(" + outputLayer->name + ") should be DetectionOutput, but was " + outputLayer->type); @@ -192,7 +185,7 @@ CNNNetwork FaceDetection::read() { slog::info << "Loading Face Detection model to the "<< deviceForInference << " device" << slog::endl; input = inputInfo.begin()->first; - return netReader.getNetwork(); + return network; } void FaceDetection::fetchResults() { @@ -300,25 +293,18 @@ AgeGenderDetection::Result AgeGenderDetection::operator[] (int idx) const { return r; } -CNNNetwork AgeGenderDetection::read() { +CNNNetwork AgeGenderDetection::read(const InferenceEngine::Core& ie) { slog::info << "Loading network files for Age/Gender Recognition network" << slog::endl; - CNNNetReader netReader; // Read network - netReader.ReadNetwork(pathToModel); - + auto network = ie.ReadNetwork(pathToModel); // Set maximum batch size to be used. - netReader.getNetwork().setBatchSize(maxBatch); - slog::info << "Batch size is set to " << netReader.getNetwork().getBatchSize() << " for Age/Gender Recognition network" << slog::endl; - - - // Extract model name and load its weights - std::string binFileName = fileNameNoExt(pathToModel) + ".bin"; - netReader.ReadWeights(binFileName); + network.setBatchSize(maxBatch); + slog::info << "Batch size is set to " << network.getBatchSize() << " for Age/Gender Recognition network" << slog::endl; // ---------------------------Check inputs ------------------------------------------------------------- // Age/Gender Recognition network should have one input and two outputs slog::info << "Checking Age/Gender Recognition network inputs" << slog::endl; - InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Age/Gender Recognition network should have only one input"); } @@ -329,7 +315,7 @@ CNNNetwork AgeGenderDetection::read() { // ---------------------------Check outputs ------------------------------------------------------------ slog::info << "Checking Age/Gender Recognition network outputs" << slog::endl; - OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 2) { throw std::logic_error("Age/Gender Recognition network should have two output layers"); } @@ -377,7 +363,7 @@ CNNNetwork AgeGenderDetection::read() { slog::info << "Loading Age/Gender Recognition model to the "<< deviceForInference << " plugin" << slog::endl; _enabled = true; - return netReader.getNetwork(); + return network; } @@ -434,21 +420,17 @@ HeadPoseDetection::Results HeadPoseDetection::operator[] (int idx) const { return r; } -CNNNetwork HeadPoseDetection::read() { +CNNNetwork HeadPoseDetection::read(const InferenceEngine::Core& ie) { slog::info << "Loading network files for Head Pose Estimation network" << slog::endl; - CNNNetReader netReader; // Read network model - netReader.ReadNetwork(pathToModel); + auto network = ie.ReadNetwork(pathToModel); // Set maximum batch size - netReader.getNetwork().setBatchSize(maxBatch); - slog::info << "Batch size is set to " << netReader.getNetwork().getBatchSize() << " for Head Pose Estimation network" << slog::endl; - // Extract model name and load its weights - std::string binFileName = fileNameNoExt(pathToModel) + ".bin"; - netReader.ReadWeights(binFileName); + network.setBatchSize(maxBatch); + slog::info << "Batch size is set to " << network.getBatchSize() << " for Head Pose Estimation network" << slog::endl; // ---------------------------Check inputs ------------------------------------------------------------- slog::info << "Checking Head Pose Estimation network inputs" << slog::endl; - InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Head Pose Estimation network should have only one input"); } @@ -459,7 +441,7 @@ CNNNetwork HeadPoseDetection::read() { // ---------------------------Check outputs ------------------------------------------------------------ slog::info << "Checking Head Pose Estimation network outputs" << slog::endl; - OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 3) { throw std::logic_error("Head Pose Estimation network should have 3 outputs"); } @@ -499,7 +481,7 @@ CNNNetwork HeadPoseDetection::read() { slog::info << "Loading Head Pose Estimation model to the "<< deviceForInference << " plugin" << slog::endl; _enabled = true; - return netReader.getNetwork(); + return network; } EmotionsDetection::EmotionsDetection(const std::string &pathToModel, @@ -538,8 +520,6 @@ void EmotionsDetection::enqueue(const cv::Mat &face) { } std::map EmotionsDetection::operator[] (int idx) const { - // Vector of supported emotions - static const std::vector emotionsVec = {"neutral", "happy", "sad", "surprise", "anger"}; auto emotionsVecSize = emotionsVec.size(); Blob::Ptr emotionsBlob = request->GetBlob(outputEmotions); @@ -578,27 +558,19 @@ std::map EmotionsDetection::operator[] (int idx) const { return emotions; } -CNNNetwork EmotionsDetection::read() { +CNNNetwork EmotionsDetection::read(const InferenceEngine::Core& ie) { slog::info << "Loading network files for Emotions Recognition" << slog::endl; - InferenceEngine::CNNNetReader netReader; // Read network model - netReader.ReadNetwork(pathToModel); - + auto network = ie.ReadNetwork(pathToModel); // Set maximum batch size - netReader.getNetwork().setBatchSize(maxBatch); - slog::info << "Batch size is set to " << netReader.getNetwork().getBatchSize() << " for Emotions Recognition" << slog::endl; - - - // Extract model name and load its weights - std::string binFileName = fileNameNoExt(pathToModel) + ".bin"; - netReader.ReadWeights(binFileName); - + network.setBatchSize(maxBatch); + slog::info << "Batch size is set to " << network.getBatchSize() << " for Emotions Recognition" << slog::endl; // ----------------------------------------------------------------------------------------------------- // Emotions Recognition network should have one input and one output. // ---------------------------Check inputs ------------------------------------------------------------- slog::info << "Checking Emotions Recognition network inputs" << slog::endl; - InferenceEngine::InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InferenceEngine::InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Emotions Recognition network should have only one input"); } @@ -609,7 +581,7 @@ CNNNetwork EmotionsDetection::read() { // ---------------------------Check outputs ------------------------------------------------------------ slog::info << "Checking Emotions Recognition network outputs" << slog::endl; - InferenceEngine::OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + InferenceEngine::OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 1) { throw std::logic_error("Emotions Recognition network should have one output layer"); } @@ -641,7 +613,7 @@ CNNNetwork EmotionsDetection::read() { slog::info << "Loading Emotions Recognition model to the "<< deviceForInference << " plugin" << slog::endl; _enabled = true; - return netReader.getNetwork(); + return network; } @@ -708,21 +680,17 @@ std::vector FacialLandmarksDetection::operator[] (int idx) const { return normedLandmarks; } -CNNNetwork FacialLandmarksDetection::read() { +CNNNetwork FacialLandmarksDetection::read(const InferenceEngine::Core& ie) { slog::info << "Loading network files for Facial Landmarks Estimation" << slog::endl; - CNNNetReader netReader; // Read network model - netReader.ReadNetwork(pathToModel); + auto network = ie.ReadNetwork(pathToModel); // Set maximum batch size - netReader.getNetwork().setBatchSize(maxBatch); - slog::info << "Batch size is set to " << netReader.getNetwork().getBatchSize() << " for Facial Landmarks Estimation network" << slog::endl; - // Extract model name and load its weights - std::string binFileName = fileNameNoExt(pathToModel) + ".bin"; - netReader.ReadWeights(binFileName); + network.setBatchSize(maxBatch); + slog::info << "Batch size is set to " << network.getBatchSize() << " for Facial Landmarks Estimation network" << slog::endl; // ---------------------------Check inputs ------------------------------------------------------------- slog::info << "Checking Facial Landmarks Estimation network inputs" << slog::endl; - InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InputsDataMap inputInfo(network.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Facial Landmarks Estimation network should have only one input"); } @@ -733,7 +701,7 @@ CNNNetwork FacialLandmarksDetection::read() { // ---------------------------Check outputs ------------------------------------------------------------ slog::info << "Checking Facial Landmarks Estimation network outputs" << slog::endl; - OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + OutputsDataMap outputInfo(network.getOutputsInfo()); if (outputInfo.size() != 1) { throw std::logic_error("Facial Landmarks Estimation network should have only one output"); } @@ -763,7 +731,7 @@ CNNNetwork FacialLandmarksDetection::read() { slog::info << "Loading Facial Landmarks Estimation model to the "<< deviceForInference << " plugin" << slog::endl; _enabled = true; - return netReader.getNetwork(); + return network; } @@ -780,7 +748,7 @@ void Load::into(InferenceEngine::Core & ie, const std::string & deviceName, bool config[PluginConfigParams::KEY_DYN_BATCH_ENABLED] = PluginConfigParams::YES; } - detector.net = ie.LoadNetwork(detector.read(), deviceName, config); + detector.net = ie.LoadNetwork(detector.read(ie), deviceName, config); } } diff --git a/demos/interactive_face_detection_demo/detectors.hpp b/demos/interactive_face_detection_demo/detectors.hpp index b36639f850f..2c52a0abdf1 100644 --- a/demos/interactive_face_detection_demo/detectors.hpp +++ b/demos/interactive_face_detection_demo/detectors.hpp @@ -45,7 +45,7 @@ struct BaseDetection { mutable bool _enabled; const bool doRawOutputMessages; - BaseDetection(std::string topoName, + BaseDetection(const std::string &topoName, const std::string &pathToModel, const std::string &deviceForInference, int maxBatch, bool isBatchDynamic, bool isAsync, @@ -54,7 +54,7 @@ struct BaseDetection { virtual ~BaseDetection(); InferenceEngine::ExecutableNetwork* operator ->(); - virtual InferenceEngine::CNNNetwork read() = 0; + virtual InferenceEngine::CNNNetwork read(const InferenceEngine::Core& ie) = 0; virtual void submitRequest(); virtual void wait(); bool enabled() const; @@ -90,7 +90,7 @@ struct FaceDetection : BaseDetection { float bb_enlarge_coefficient, float bb_dx_coefficient, float bb_dy_coefficient); - InferenceEngine::CNNNetwork read() override; + InferenceEngine::CNNNetwork read(const InferenceEngine::Core& ie) override; void submitRequest() override; void enqueue(const cv::Mat &frame); @@ -113,7 +113,7 @@ struct AgeGenderDetection : BaseDetection { int maxBatch, bool isBatchDynamic, bool isAsync, bool doRawOutputMessages); - InferenceEngine::CNNNetwork read() override; + InferenceEngine::CNNNetwork read(const InferenceEngine::Core& ie) override; void submitRequest() override; void enqueue(const cv::Mat &face); @@ -139,7 +139,7 @@ struct HeadPoseDetection : BaseDetection { int maxBatch, bool isBatchDynamic, bool isAsync, bool doRawOutputMessages); - InferenceEngine::CNNNetwork read() override; + InferenceEngine::CNNNetwork read(const InferenceEngine::Core& ie) override; void submitRequest() override; void enqueue(const cv::Mat &face); @@ -156,7 +156,7 @@ struct EmotionsDetection : BaseDetection { int maxBatch, bool isBatchDynamic, bool isAsync, bool doRawOutputMessages); - InferenceEngine::CNNNetwork read() override; + InferenceEngine::CNNNetwork read(const InferenceEngine::Core& ie) override; void submitRequest() override; void enqueue(const cv::Mat &face); @@ -177,7 +177,7 @@ struct FacialLandmarksDetection : BaseDetection { int maxBatch, bool isBatchDynamic, bool isAsync, bool doRawOutputMessages); - InferenceEngine::CNNNetwork read() override; + InferenceEngine::CNNNetwork read(const InferenceEngine::Core& ie) override; void submitRequest() override; void enqueue(const cv::Mat &face); diff --git a/demos/interactive_face_detection_demo/interactive_face_detection.hpp b/demos/interactive_face_detection_demo/interactive_face_detection.hpp index bae31f48a16..98d038e87e6 100644 --- a/demos/interactive_face_detection_demo/interactive_face_detection.hpp +++ b/demos/interactive_face_detection_demo/interactive_face_detection.hpp @@ -10,254 +10,99 @@ #include #include -/// @brief Message for help argument static const char help_message[] = "Print a usage message"; - -/// @brief Message for images argument static const char input_video_message[] = "Required. Path to a video file (specify \"cam\" to work with camera)."; - -/// @brief Message for images argument static const char output_video_message[] = "Optional. Path to an output video file."; - -/// @brief message for model argument static const char face_detection_model_message[] = "Required. Path to an .xml file with a trained Face Detection model."; static const char age_gender_model_message[] = "Optional. Path to an .xml file with a trained Age/Gender Recognition model."; static const char head_pose_model_message[] = "Optional. Path to an .xml file with a trained Head Pose Estimation model."; static const char emotions_model_message[] = "Optional. Path to an .xml file with a trained Emotions Recognition model."; static const char facial_landmarks_model_message[] = "Optional. Path to an .xml file with a trained Facial Landmarks Estimation model."; - -/// @brief Message for plugin argument -static const char plugin_message[] = "Plugin name. For example, CPU. If this parameter is specified, " \ -"the demo will look for this plugin only."; - -/// @brief Message for assigning face detection calculation to device -static const char target_device_message[] = "Optional. Target device for Face Detection network (the list of available devices is shown below). " \ -"Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device."; - -/// @brief Message for assigning age/gender calculation to device -static const char target_device_message_ag[] = "Optional. Target device for Age/Gender Recognition network (the list of available devices is shown below). " \ -"Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device."; - -/// @brief Message for assigning head pose calculation to device -static const char target_device_message_hp[] = "Optional. Target device for Head Pose Estimation network (the list of available devices is shown below). " \ -"Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device."; - -/// @brief Message for assigning emotions calculation to device -static const char target_device_message_em[] = "Optional. Target device for Emotions Recognition network (the list of available devices is shown below). " \ -"Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device."; - -/// @brief Message for assigning Facial Landmarks Estimation network to device -static const char target_device_message_lm[] = "Optional. Target device for Facial Landmarks Estimation network " \ -"(the list of available devices is shown below). Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for device specified."; - -/// @brief Message for the maximum number of simultaneously processed faces for Age Gender network -static const char num_batch_ag_message[] = "Optional. Number of maximum simultaneously processed faces for Age/Gender Recognition network " \ -"(by default, it is 16)"; - -/// @brief Message for the maximum number of simultaneously processed faces for Head Pose network -static const char num_batch_hp_message[] = "Optional. Number of maximum simultaneously processed faces for Head Pose Estimation network " \ -"(by default, it is 16)"; - -/// @brief Message for the maximum number of simultaneously processed faces for Emotions network -static const char num_batch_em_message[] = "Optional. Number of maximum simultaneously processed faces for Emotions Recognition network " \ -"(by default, it is 16)"; - -/// @brief Message for the maximum number of simultaneously processed faces for Facial Landmarks Estimation network -static const char num_batch_lm_message[] = "Optional. Number of maximum simultaneously processed faces for Facial Landmarks Estimation network " \ -"(by default, it is 16)"; - -/// @brief Message for dynamic batching support for AgeGender net +static const char plugin_message[] = "Plugin name. For example, CPU. If this parameter is specified, " + "the demo will look for this plugin only."; +static const char target_device_message[] = "Optional. Target device for Face Detection network (the list of available devices is shown below). " + "Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device."; +static const char target_device_message_ag[] = "Optional. Target device for Age/Gender Recognition network (the list of available devices is shown below). " + "Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device."; +static const char target_device_message_hp[] = "Optional. Target device for Head Pose Estimation network (the list of available devices is shown below). " + "Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device."; +static const char target_device_message_em[] = "Optional. Target device for Emotions Recognition network (the list of available devices is shown below). " + "Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device."; +static const char target_device_message_lm[] = "Optional. Target device for Facial Landmarks Estimation network " + "(the list of available devices is shown below). Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for device specified."; +static const char num_batch_ag_message[] = "Optional. Number of maximum simultaneously processed faces for Age/Gender Recognition network " + "(by default, it is 16)"; +static const char num_batch_hp_message[] = "Optional. Number of maximum simultaneously processed faces for Head Pose Estimation network " + "(by default, it is 16)"; +static const char num_batch_em_message[] = "Optional. Number of maximum simultaneously processed faces for Emotions Recognition network " + "(by default, it is 16)"; +static const char num_batch_lm_message[] = "Optional. Number of maximum simultaneously processed faces for Facial Landmarks Estimation network " + "(by default, it is 16)"; static const char dyn_batch_ag_message[] = "Optional. Enable dynamic batch size for Age/Gender Recognition network"; - -/// @brief Message for dynamic batching support for HeadPose net static const char dyn_batch_hp_message[] = "Optional. Enable dynamic batch size for Head Pose Estimation network"; - -/// @brief Message for dynamic batching support for Emotions net static const char dyn_batch_em_message[] = "Optional. Enable dynamic batch size for Emotions Recognition network"; - -/// @brief Message for dynamic batching support for Facial Landmarks Estimation network static const char dyn_batch_lm_message[] = "Optional. Enable dynamic batch size for Facial Landmarks Estimation network"; - -/// @brief Message for performance counters static const char performance_counter_message[] = "Optional. Enable per-layer performance report"; - -/// @brief Message for GPU custom kernels description -static const char custom_cldnn_message[] = "Required for GPU custom kernels. "\ -"Absolute path to an .xml file with the kernels description."; - -/// @brief Message for user library argument -static const char custom_cpu_library_message[] = "Required for CPU custom layers. " \ -"Absolute path to a shared library with the kernels implementation."; - -/// @brief Message for probability threshold argument +static const char custom_cldnn_message[] = "Required for GPU custom kernels. " + "Absolute path to an .xml file with the kernels description."; +static const char custom_cpu_library_message[] = "Required for CPU custom layers. " + "Absolute path to a shared library with the kernels implementation."; static const char thresh_output_message[] = "Optional. Probability threshold for detections"; - -/// @brief Message for face enlarge coefficient argument static const char bb_enlarge_coef_output_message[] = "Optional. Coefficient to enlarge/reduce the size of the bounding box around the detected face"; - -/// @brief Message raw output flag static const char raw_output_message[] = "Optional. Output inference results as raw values"; - -/// @brief Message do not wait for keypress after input stream completed static const char no_wait_for_keypress_message[] = "Optional. Do not wait for key press in the end."; - -/// @brief Message do not show processed video static const char no_show_processed_video[] = "Optional. Do not show processed video."; - -/// @brief Message for asynchronous mode static const char async_message[] = "Optional. Enable asynchronous mode"; - -/// @brief Message for shifting coefficient by dx for detected faces static const char dx_coef_output_message[] = "Optional. Coefficient to shift the bounding box around the detected face along the Ox axis"; - -/// @brief Message for shifting coefficient by dy for detected faces static const char dy_coef_output_message[] = "Optional. Coefficient to shift the bounding box around the detected face along the Oy axis"; - -/// @brief Message for fps argument static const char fps_output_message[] = "Optional. Maximum FPS for playing video"; - -/// @brief Message for looping video argument static const char loop_video_output_message[] = "Optional. Enable playing video on a loop"; - -/// @brief Message for smooth argument static const char no_smooth_output_message[] = "Optional. Do not smooth person attributes"; - -/// @brief Message for smooth argument static const char no_show_emotion_bar_message[] = "Optional. Do not show emotion bar"; +static const char utilization_monitors_message[] = "Optional. List of monitors to show initially."; -/// \brief Define flag for showing help message
DEFINE_bool(h, false, help_message); - -/// \brief Define parameter for set image file
-/// It is a required parameter DEFINE_string(i, "", input_video_message); - -/// \brief Define parameter for an output video file
-/// It is an optional parameter DEFINE_string(o, "", output_video_message); - -/// \brief Define parameter for Face Detection model file
-/// It is a required parameter DEFINE_string(m, "", face_detection_model_message); - -/// \brief Define parameter for Age Gender Recognition model file
-/// It is a optional parameter DEFINE_string(m_ag, "", age_gender_model_message); - -/// \brief Define parameter for Head Pose Estimation model file
-/// It is a optional parameter DEFINE_string(m_hp, "", head_pose_model_message); - -/// \brief Define parameter for Emotions Recognition model file
-/// It is a optional parameter DEFINE_string(m_em, "", emotions_model_message); - -/// \brief Define parameter for Facial Landmarks Estimation model file
-/// It is an optional parameter DEFINE_string(m_lm, "", facial_landmarks_model_message); - -/// \brief target device for Face Detection network
DEFINE_string(d, "CPU", target_device_message); - -/// \brief Define parameter for target device for Age/Gender Recognition network
DEFINE_string(d_ag, "CPU", target_device_message_ag); - -/// \brief Define parameter for target device for Head Pose Estimation network
DEFINE_string(d_hp, "CPU", target_device_message_hp); - -/// \brief Define parameter for target device for Emotions Recognition network
DEFINE_string(d_em, "CPU", target_device_message_em); - -/// \brief Define parameter for target device for Facial Landmarks Estimation network
DEFINE_string(d_lm, "CPU", target_device_message_lm); - -/// \brief Define parameter for maximum batch size for Age/Gender Recognition network
DEFINE_uint32(n_ag, 16, num_batch_ag_message); - -/// \brief Define parameter to enable dynamic batch size for Age/Gender Recognition network
DEFINE_bool(dyn_ag, false, dyn_batch_ag_message); - -/// \brief Define parameter for maximum batch size for Head Pose Estimation network
DEFINE_uint32(n_hp, 16, num_batch_hp_message); - -/// \brief Define parameter to enable dynamic batch size for Head Pose Estimation network
DEFINE_bool(dyn_hp, false, dyn_batch_hp_message); - -/// \brief Define parameter for maximum batch size for Emotions Recognition network
DEFINE_uint32(n_em, 16, num_batch_em_message); - -/// \brief Define parameter to enable dynamic batch size for Emotions Recognition network
DEFINE_bool(dyn_em, false, dyn_batch_em_message); - -/// \brief Define parameter for maximum batch size for Facial Landmarks Estimation network
DEFINE_uint32(n_lm, 16, num_batch_em_message); - -/// \brief Define parameter to enable dynamic batch size for Facial Landmarks Estimation network
DEFINE_bool(dyn_lm, false, dyn_batch_em_message); - -/// \brief Define parameter to enable per-layer performance report
DEFINE_bool(pc, false, performance_counter_message); - -/// @brief Define parameter for GPU custom kernels path
-/// Default is ./lib DEFINE_string(c, "", custom_cldnn_message); - -/// @brief Define parameter for absolute path to CPU library with user layers
-/// It is an optional parameter DEFINE_string(l, "", custom_cpu_library_message); - -/// \brief Define a flag to output raw scoring results
-/// It is an optional parameter DEFINE_bool(r, false, raw_output_message); - -/// \brief Define a parameter for probability threshold for detections
-/// It is an optional parameter DEFINE_double(t, 0.5, thresh_output_message); - -/// \brief Define a parameter to enlarge the bounding box around the detected face for more robust operation of face analytics networks
-/// It is an optional parameter DEFINE_double(bb_enlarge_coef, 1.2, bb_enlarge_coef_output_message); - -/// \brief Define a flag to disable keypress exit
-/// It is an optional parameter DEFINE_bool(no_wait, false, no_wait_for_keypress_message); - -/// \brief Define a flag to disable showing processed video
-/// It is an optional parameter DEFINE_bool(no_show, false, no_show_processed_video); - -/// \brief Define a flag to enable aynchronous execution
-/// It is an optional parameter DEFINE_bool(async, false, async_message); - -/// \brief Define a parameter to shift face bounding box by Ox for more robust operation of face analytics networks
-/// It is an optional parameter DEFINE_double(dx_coef, 1, dx_coef_output_message); - -/// \brief Define a parameter to shift face bounding box by Oy for more robust operation of face analytics networks
-/// It is an optional parameter DEFINE_double(dy_coef, 1, dy_coef_output_message); - -/// \brief Define a parameter to play video with defined fps
-/// It is an optional parameter DEFINE_double(fps, -1, fps_output_message); - -/// \brief Define a flag to loop video
-/// It is an optional parameter DEFINE_bool(loop_video, false, loop_video_output_message); - -/// \brief Define a flag to disable smoothing person attributes
-/// It is an optional parameter DEFINE_bool(no_smooth, false, no_smooth_output_message); - -/// \brief Define a flag to disable showing emotion bar
-/// It is an optional parameter DEFINE_bool(no_show_emotion_bar, false, no_show_emotion_bar_message); +DEFINE_string(u, "", utilization_monitors_message); /** @@ -306,4 +151,5 @@ static void showUsage() { std::cout << " -loop_video " << loop_video_output_message << std::endl; std::cout << " -no_smooth " << no_smooth_output_message << std::endl; std::cout << " -no_show_emotion_bar " << no_show_emotion_bar_message << std::endl; + std::cout << " -u " << utilization_monitors_message << std::endl; } diff --git a/demos/interactive_face_detection_demo/main.cpp b/demos/interactive_face_detection_demo/main.cpp index 332a07e1bd3..e77e79fc1b9 100644 --- a/demos/interactive_face_detection_demo/main.cpp +++ b/demos/interactive_face_detection_demo/main.cpp @@ -25,6 +25,7 @@ #include +#include #include #include @@ -34,9 +35,6 @@ #include "visualizer.hpp" #include -#ifdef WITH_EXTENSIONS -#include -#endif using namespace InferenceEngine; @@ -108,7 +106,7 @@ int main(int argc, char *argv[]) { Core ie; std::set loadedDevices; - std::vector> cmdOptions = { + std::pair cmdOptions[] = { {FLAGS_d, FLAGS_m}, {FLAGS_d_ag, FLAGS_m_ag}, {FLAGS_d_hp, FLAGS_m_hp}, @@ -138,9 +136,6 @@ int main(int argc, char *argv[]) { /** Loading extensions for the CPU device **/ if ((deviceName.find("CPU") != std::string::npos)) { -#ifdef WITH_EXTENSIONS - ie.AddExtension(std::make_shared(), "CPU"); -#endif if (!FLAGS_l.empty()) { // CPU(MKLDNN) extensions are loaded as a shared library and passed as a pointer to base extension @@ -180,8 +175,6 @@ int main(int argc, char *argv[]) { std::ostringstream out; size_t framesCounter = 0; - bool frameReadStatus; - bool isLastFrame; int delay = 1; double msrate = -1; cv::Mat prev_frame, next_frame; @@ -207,18 +200,23 @@ int main(int argc, char *argv[]) { prev_frame = frame.clone(); // Reading the next frame - frameReadStatus = cap.read(frame); + bool frameReadStatus = cap.read(frame); std::cout << "To close the application, press 'CTRL+C' here"; if (!FLAGS_no_show) { - std::cout << " or switch to the output window and press any key"; + std::cout << " or switch to the output window and press Q or Esc"; } std::cout << std::endl; + const cv::Point THROUGHPUT_METRIC_POSITION{10, 45}; + + cv::Size graphSize{static_cast(cap.get(cv::CAP_PROP_FRAME_WIDTH) / 4), 60}; + Presenter presenter(FLAGS_u, THROUGHPUT_METRIC_POSITION.y + 15, graphSize); + while (true) { timer.start("total"); framesCounter++; - isLastFrame = !frameReadStatus; + bool isLastFrame = !frameReadStatus; // Retrieving face detection results for the previous frame faceDetector.wait(); @@ -289,7 +287,7 @@ int main(int argc, char *argv[]) { float intensity_mean = calcMean(prev_frame(rect)); if ((face == nullptr) || - ((face != nullptr) && ((std::abs(intensity_mean - face->_intensity_mean) / face->_intensity_mean) > 0.07f))) { + ((std::abs(intensity_mean - face->_intensity_mean) / face->_intensity_mean) > 0.07f)) { face = std::make_shared(id++, rect); } else { prev_faces.remove(face); @@ -330,12 +328,14 @@ int main(int argc, char *argv[]) { faces.push_back(face); } + presenter.drawGraphs(prev_frame); + // Visualizing results if (!FLAGS_no_show || !FLAGS_o.empty()) { out.str(""); out << "Total image throughput: " << std::fixed << std::setprecision(2) << 1000.f / (timer["total"].getSmoothedDuration()) << " fps"; - cv::putText(prev_frame, out.str(), cv::Point2f(10, 45), cv::FONT_HERSHEY_TRIPLEX, 1.2, + cv::putText(prev_frame, out.str(), THROUGHPUT_METRIC_POSITION, cv::FONT_HERSHEY_TRIPLEX, 1, cv::Scalar(255, 0, 0), 2); // drawing faces @@ -367,8 +367,12 @@ int main(int argc, char *argv[]) { cv::waitKey(0); } break; - } else if (!FLAGS_no_show && -1 != cv::waitKey(delay)) { - break; + } else if (!FLAGS_no_show) { + int key = cv::waitKey(delay); + if (27 == key || 'Q' == key || 'q' == key) { + break; + } + presenter.handleKey(key); } } @@ -383,6 +387,8 @@ int main(int argc, char *argv[]) { emotionsDetector.printPerformanceCounts(getFullDeviceName(ie, FLAGS_d_em)); facialLandmarksDetector.printPerformanceCounts(getFullDeviceName(ie, FLAGS_d_lm)); } + + std::cout << presenter.reportMeans() << '\n'; // --------------------------------------------------------------------------------------------------- if (!FLAGS_o.empty()) { diff --git a/demos/mask_rcnn_demo/CMakeLists.txt b/demos/mask_rcnn_demo/CMakeLists.txt index 872e3c4f363..2d8cb775ce1 100644 --- a/demos/mask_rcnn_demo/CMakeLists.txt +++ b/demos/mask_rcnn_demo/CMakeLists.txt @@ -5,5 +5,4 @@ ie_add_sample(NAME mask_rcnn_demo SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/main.cpp" HEADERS "${CMAKE_CURRENT_SOURCE_DIR}/mask_rcnn_demo.h" - DEPENDENCIES format_reader OPENCV_DEPENDENCIES imgcodecs) diff --git a/demos/mask_rcnn_demo/README.md b/demos/mask_rcnn_demo/README.md index e43fa4f1ec9..b8cdcfa88a2 100644 --- a/demos/mask_rcnn_demo/README.md +++ b/demos/mask_rcnn_demo/README.md @@ -13,7 +13,7 @@ Upon the start-up, the demo application reads command line parameters and loads ## Running Running the application with the `-h` option yields the following usage message: -```sh +``` ./mask_rcnn_demo -h InferenceEngine: API version ............ @@ -23,13 +23,13 @@ mask_rcnn_demo [OPTION] Options: -h Print a usage message. - -i "" Required. Path to an .bmp image. + -i "" Required. Path to a .bmp image. -m "" Required. Path to an .xml file with a trained model. -l "" Required for CPU custom layers. Absolute path to a shared library with the kernels implementations. Or -c "" Required for GPU custom kernels. Absolute path to the .xml file with the kernels descriptions. - -d "" Optional. Specify the target device to infer on (the list of available devices is shown below). Use "-d HETERO:" format to specify HETERO plugin. The demo will look for a suitable plugin for a specified device (CPU by default). - -detection_output_name "" Optional. The name of detection output layer. Default value is "detection_output" + -d "" Optional. Specify the target device to infer on (the list of available devices is shown below). Use "-d HETERO:" format to specify HETERO plugin. The demo will look for a suitable plugin for a specified device (CPU by default) + -detection_output_name "" Optional. The name of detection output layer. Default value is "reshape_do_2d" -masks_name "" Optional. The name of masks layer. Default value is "masks" ``` @@ -41,7 +41,7 @@ To run the demo, you can use public or pre-trained models. To download the pre-t You can use the following command to do inference on CPU on an image using a trained network: ```sh -./mask_rcnn_demo -i /inputImage.bmp -m /faster_rcnn.xml +./mask_rcnn_demo -i /inputImage.bmp -m /mask_rcnn_inception_resnet_v2_atrous_coco.xml ``` ## Demo Output diff --git a/demos/mask_rcnn_demo/main.cpp b/demos/mask_rcnn_demo/main.cpp index 35728359ea6..ea1abc4c96a 100644 --- a/demos/mask_rcnn_demo/main.cpp +++ b/demos/mask_rcnn_demo/main.cpp @@ -17,11 +17,6 @@ #include #include -#ifdef WITH_EXTENSIONS -#include -#endif - -#include #include #include @@ -63,27 +58,15 @@ int main(int argc, char *argv[]) { } /** This vector stores paths to the processed images **/ - std::vector images; - parseInputFilesArguments(images); - if (images.empty()) throw std::logic_error("No suitable images were found"); + std::vector imagePaths; + parseInputFilesArguments(imagePaths); + if (imagePaths.empty()) throw std::logic_error("No suitable images were found"); // ----------------------------------------------------------------------------------------------------- // ---------------------Load inference engine------------------------------------------------ slog::info << "Loading Inference Engine" << slog::endl; Core ie; -#ifdef WITH_EXTENSIONS - /** Loading default extensions **/ - if (FLAGS_d.find("CPU") != std::string::npos) { - /** - * cpu_extensions library is compiled from "extension" folder containing - * custom MKLDNNPlugin layer implementations. These layers are not supported - * by mkldnn, but they can be useful for inferring custom topologies. - **/ - ie.AddExtension(std::make_shared(), "CPU"); - } -#endif - if (!FLAGS_l.empty()) { // CPU(MKLDNN) extensions are loaded as a shared library and passed as a pointer to base extension auto extension_ptr = make_so_pointer(FLAGS_l); @@ -105,14 +88,8 @@ int main(int argc, char *argv[]) { // --------------------Load network (Generated xml/bin files)------------------------------------------- slog::info << "Loading network files" << slog::endl; - InferenceEngine::CNNNetReader networkReader; /** Read network model **/ - networkReader.ReadNetwork(FLAGS_m); - - /** Extract model name and load weights **/ - std::string binFileName = fileNameNoExt(FLAGS_m) + ".bin"; - networkReader.ReadWeights(binFileName); - auto network = networkReader.getNetwork(); + auto network = ie.ReadNetwork(FLAGS_m); // add DetectionOutput layer as output so we can get detected boxes and their probabilities network.addOutput(FLAGS_detection_output_name.c_str(), 0); @@ -141,44 +118,38 @@ int main(int argc, char *argv[]) { const TensorDesc& inputDesc = inputInfo[imageInputName]->getTensorDesc(); IE_ASSERT(inputDesc.getDims().size() == 4); size_t netBatchSize = getTensorBatch(inputDesc); - size_t netInputChannels = getTensorChannels(inputDesc); size_t netInputHeight = getTensorHeight(inputDesc); size_t netInputWidth = getTensorWidth(inputDesc); slog::info << "Network batch size is " << netBatchSize << slog::endl; - /** Collect images data ptrs **/ - std::vector> imagesData; - std::vector images_cv; + /** Collect images **/ + std::vector images; - if (netBatchSize > images.size()) { - slog::warn << "Network batch size is greater than number of images (" << images.size() << + if (netBatchSize > imagePaths.size()) { + slog::warn << "Network batch size is greater than number of images (" << imagePaths.size() << "), some input files will be duplicated" << slog::endl; - } else if (netBatchSize < images.size()) { - slog::warn << "Network batch size is less than number of images (" << images.size() << + } else if (netBatchSize < imagePaths.size()) { + slog::warn << "Network batch size is less than number of images (" << imagePaths.size() << "), some input files will be ignored" << slog::endl; } for (size_t i = 0, inputIndex = 0; i < netBatchSize; i++, inputIndex++) { - if (inputIndex >= images.size()) { + if (inputIndex >= imagePaths.size()) { inputIndex = 0; } - slog::info << "Prepare image " << images[inputIndex] << slog::endl; + slog::info << "Prepare image " << imagePaths[inputIndex] << slog::endl; - images_cv.push_back(cv::imread(images[inputIndex], cv::IMREAD_COLOR)); + cv::Mat image = cv::imread(imagePaths[inputIndex], cv::IMREAD_COLOR); - FormatReader::ReaderPtr reader(images[inputIndex].c_str()); - if (reader.get() == nullptr) { - slog::warn << "Image " + images[inputIndex] + " cannot be read!" << slog::endl; + if (image.empty()) { + slog::warn << "Image " + imagePaths[inputIndex] + " cannot be read!" << slog::endl; continue; } - /** Getting image data **/ - std::shared_ptr data(reader->getData(netInputWidth, netInputHeight)); - if (data != nullptr) { - imagesData.push_back(data); - } + + images.push_back(image); } - if (imagesData.empty()) throw std::logic_error("Valid input images were not found!"); + if (images.empty()) throw std::logic_error("Valid input images were not found!"); // ----------------------------------------------------------------------------------------------------- @@ -211,21 +182,9 @@ int main(int argc, char *argv[]) { /** Fill first input tensor with images. First b channel, then g and r channels **/ if (inputInfoItem.second->getTensorDesc().getDims().size() == 4) { - auto data = input->buffer().as::value_type *>(); - size_t image_size = netInputHeight * netInputWidth; - /** Iterate over all input images **/ - for (size_t image_id = 0; image_id < imagesData.size(); ++image_id) { - /** Iterate over all pixels in image (b,g,r) **/ - for (size_t pid = 0; pid < image_size; pid++) { - /** Iterate over all channels **/ - for (size_t ch = 0; ch < netInputChannels; ++ch) { - /** [images stride + channels stride + pixel id ] all in bytes **/ - data[image_id * image_size * netInputChannels + ch * image_size + pid] = imagesData.at( - image_id).get()[pid * netInputChannels + ch]; - } - } - } + for (size_t image_id = 0; image_id < images.size(); ++image_id) + matU8ToBlob(images[image_id], input, image_id); } /** Fill second input tensor with image info **/ @@ -257,7 +216,7 @@ int main(int argc, char *argv[]) { const float PROBABILITY_THRESHOLD = 0.2f; const float MASK_THRESHOLD = 0.5f; // threshold used to determine whether mask pixel corresponds to object or to background // amount of elements in each detected box description (batch, label, prob, x1, y1, x2, y2) - IE_ASSERT(do_blob->getTensorDesc().getDims().size() == 4); + IE_ASSERT(do_blob->getTensorDesc().getDims().size() == 2); size_t BOX_DESCRIPTION_SIZE = do_blob->getTensorDesc().getDims().back(); const TensorDesc& masksDesc = masks_blob->getTensorDesc(); @@ -270,34 +229,10 @@ int main(int argc, char *argv[]) { size_t box_stride = W * H * C; - // some colours - std::vector> colors = { - {128, 64, 128}, - {232, 35, 244}, - {70, 70, 70}, - {156, 102, 102}, - {153, 153, 190}, - {153, 153, 153}, - {30, 170, 250}, - {0, 220, 220}, - {35, 142, 107}, - {152, 251, 152}, - {180, 130, 70}, - {60, 20, 220}, - {0, 0, 255}, - {142, 0, 0}, - {70, 0, 0}, - {100, 60, 0}, - {90, 0, 0}, - {230, 0, 0}, - {32, 11, 119}, - {0, 74, 111}, - {81, 0, 81} - }; std::map class_color; std::vector output_images; - for (const auto &img : images_cv) { + for (const auto &img : images) { output_images.push_back(img.clone()); } @@ -310,17 +245,16 @@ int main(int argc, char *argv[]) { if (batch >= static_cast(netBatchSize)) throw std::logic_error("Invalid batch ID within detection output box"); float prob = box_info[2]; - float x1 = std::min(std::max(0.0f, box_info[3] * images_cv[batch].size().width), static_cast(images_cv[batch].size().width)); - float y1 = std::min(std::max(0.0f, box_info[4] * images_cv[batch].size().height), static_cast(images_cv[batch].size().height)); - float x2 = std::min(std::max(0.0f, box_info[5] * images_cv[batch].size().width), static_cast(images_cv[batch].size().width)); - float y2 = std::min(std::max(0.0f, box_info[6] * images_cv[batch].size().height), static_cast(images_cv[batch].size().height)); - int box_width = std::min(static_cast(std::max(0.0f, x2 - x1)), images_cv[batch].size().width); - int box_height = std::min(static_cast(std::max(0.0f, y2 - y1)), images_cv[batch].size().height); + float x1 = std::min(std::max(0.0f, box_info[3] * images[batch].cols), static_cast(images[batch].cols)); + float y1 = std::min(std::max(0.0f, box_info[4] * images[batch].rows), static_cast(images[batch].rows)); + float x2 = std::min(std::max(0.0f, box_info[5] * images[batch].cols), static_cast(images[batch].cols)); + float y2 = std::min(std::max(0.0f, box_info[6] * images[batch].rows), static_cast(images[batch].rows)); + int box_width = std::min(static_cast(std::max(0.0f, x2 - x1)), images[batch].cols); + int box_height = std::min(static_cast(std::max(0.0f, y2 - y1)), images[batch].rows); auto class_id = static_cast(box_info[1] + 1e-6f); if (prob > PROBABILITY_THRESHOLD) { - if (class_color.find(class_id) == class_color.end()) - class_color[class_id] = class_color.size(); - auto& color = colors[class_color[class_id]]; + size_t color_index = class_color.emplace(class_id, class_color.size()).first->second; + auto& color = CITYSCAPES_COLORS[color_index % arraySize(CITYSCAPES_COLORS)]; float* mask_arr = masks_data + box_stride * box + H * W * (class_id - 1); slog::info << "Detected class " << class_id << " with probability " << prob << " from batch " << batch << ": [" << x1 << ", " << y1 << "], [" << x2 << ", " << y2 << "]" << slog::endl; @@ -333,13 +267,9 @@ int main(int argc, char *argv[]) { cv::Mat resized_mask_mat(box_height, box_width, CV_32FC1); cv::resize(mask_mat, resized_mask_mat, cv::Size(box_width, box_height)); - cv::Mat uchar_resized_mask(box_height, box_width, images_cv[batch].type()); - - for (int h = 0; h < resized_mask_mat.size().height; ++h) - for (int w = 0; w < resized_mask_mat.size().width; ++w) - for (int ch = 0; ch < uchar_resized_mask.channels(); ++ch) - uchar_resized_mask.at(h, w)[ch] = resized_mask_mat.at(h, w) > MASK_THRESHOLD ? - 255 * color[ch]: roi_input_img.at(h, w)[ch]; + cv::Mat uchar_resized_mask(box_height, box_width, CV_8UC3, + cv::Scalar(color.blue(), color.green(), color.red())); + roi_input_img.copyTo(uchar_resized_mask, resized_mask_mat <= MASK_THRESHOLD); cv::addWeighted(uchar_resized_mask, alpha, roi_input_img, 1.0f - alpha, 0.0f, roi_input_img); cv::rectangle(output_images[batch], roi, cv::Scalar(0, 0, 1), 1); diff --git a/demos/mask_rcnn_demo/mask_rcnn_demo.h b/demos/mask_rcnn_demo/mask_rcnn_demo.h index af05f0c536b..5a30909e649 100644 --- a/demos/mask_rcnn_demo/mask_rcnn_demo.h +++ b/demos/mask_rcnn_demo/mask_rcnn_demo.h @@ -9,60 +9,26 @@ #include #include -/// @brief message for help argument static const char help_message[] = "Print a usage message."; - -/// @brief message for images argument -static const char image_message[] = "Required. Path to an .bmp image."; - -/// @brief message for model argument -static const char model_message[] = "Required. Path to an .xml file with a trained model.";\ - -/// @brief message for assigning cnn calculation to device -static const char target_device_message[] = "Optional. Specify the target device to infer on (the list of available devices is shown below). " \ -"Use \"-d HETERO:\" format to specify HETERO plugin. " \ -"The demo will look for a suitable plugin for a specified device (CPU by default)"; - -/// @brief message for clDNN custom kernels desc -static const char custom_cldnn_message[] = "Required for GPU custom kernels. "\ -"Absolute path to the .xml file with the kernels descriptions."; - -/// @brief message for user library argument -static const char custom_cpu_library_message[] = "Required for CPU custom layers. " \ -"Absolute path to a shared library with the kernels implementations."; - -/// @brief message for detection output layer name argument -static const char detection_output_layer_name_message[] = "Optional. The name of detection output layer. Default value is \"detection_output\""; - -/// @brief message for masks layer name argument +static const char image_message[] = "Required. Path to a .bmp image."; +static const char model_message[] = "Required. Path to an .xml file with a trained model."; +static const char target_device_message[] = "Optional. Specify the target device to infer on (the list of available devices is shown below). " + "Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo will look for a suitable plugin for a specified device (CPU by default)"; +static const char custom_cldnn_message[] = "Required for GPU custom kernels. " + "Absolute path to the .xml file with the kernels descriptions."; +static const char custom_cpu_library_message[] = "Required for CPU custom layers. " + "Absolute path to a shared library with the kernels implementations."; +static const char detection_output_layer_name_message[] = "Optional. The name of detection output layer. Default value is \"reshape_do_2d\""; static const char masks_layer_name_message[] = "Optional. The name of masks layer. Default value is \"masks\""; -/// @brief Define parameter for clDNN custom kernels path
-/// Default is ./lib DEFINE_string(c, "", custom_cldnn_message); - -/// @brief Absolute path to CPU library with user layers
-/// It is a optional parameter DEFINE_string(l, "", custom_cpu_library_message); - -/// @brief Define flag for showing help message
DEFINE_bool(h, false, help_message); - -/// @brief Define parameter for set image file
-/// It is a required parameter DEFINE_string(i, "", image_message); - -/// @brief Define parameter for set model file
-/// It is a required parameter DEFINE_string(m, "", model_message); - -/// @brief device the target device to infer on
DEFINE_string(d, "CPU", target_device_message); - -/// @brief Custom Detection Output layer name -DEFINE_string(detection_output_name, "detection_output", detection_output_layer_name_message); - -/// @brief Custom layer name producing masks +DEFINE_string(detection_output_name, "reshape_do_2d", detection_output_layer_name_message); DEFINE_string(masks_name, "masks", masks_layer_name_message); /** diff --git a/demos/mask_rcnn_demo/models.lst b/demos/mask_rcnn_demo/models.lst new file mode 100644 index 00000000000..1ffb9148c4f --- /dev/null +++ b/demos/mask_rcnn_demo/models.lst @@ -0,0 +1,5 @@ +# This file can be used with the --list option of the model downloader. +mask_rcnn_inception_resnet_v2_atrous_coco +mask_rcnn_inception_v2_coco +mask_rcnn_resnet101_atrous_coco +mask_rcnn_resnet50_atrous_coco diff --git a/demos/multichannel_demo/CMakeLists.txt b/demos/multi_channel/CMakeLists.txt similarity index 87% rename from demos/multichannel_demo/CMakeLists.txt rename to demos/multi_channel/CMakeLists.txt index ff575b7784f..5e00db8ae1e 100644 --- a/demos/multichannel_demo/CMakeLists.txt +++ b/demos/multi_channel/CMakeLists.txt @@ -21,5 +21,6 @@ if(MULTICHANNEL_DEMO_USE_NATIVE_CAM) endif() add_subdirectory(common) -add_subdirectory(fd) -add_subdirectory(hpe) +add_subdirectory(face_detection_demo) +add_subdirectory(human_pose_estimation_demo) +add_subdirectory(object_detection_demo_yolov3) diff --git a/demos/multi_channel/README.md b/demos/multi_channel/README.md new file mode 100644 index 00000000000..3876b9d5a6a --- /dev/null +++ b/demos/multi_channel/README.md @@ -0,0 +1,6 @@ +# Multi-Channel C++ Demos + +The demos provide an inference pipeline for three multi-channel scenarios: face detection, human pose estimation and object detection yolov3. For more information, refer to the corresponding pages: +* [Multi-Channel Face Detection C++ Demo](./face_detection_demo/README.md) +* [Multi-Channel Human Pose Estimation C++ Demo](./human_pose_estimation_demo/README.md) +* [Multi-Channel Object Detection Yolov3 C++ Demo](./object_detection_demo_yolov3/README.md) diff --git a/demos/multichannel_demo/common/CMakeLists.txt b/demos/multi_channel/common/CMakeLists.txt similarity index 95% rename from demos/multichannel_demo/common/CMakeLists.txt rename to demos/multi_channel/common/CMakeLists.txt index 740f3e1ec46..d5e5d933e40 100644 --- a/demos/multichannel_demo/common/CMakeLists.txt +++ b/demos/multi_channel/common/CMakeLists.txt @@ -110,13 +110,7 @@ endif() target_include_directories(${TARGET_NAME} PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}") -if(TARGET IE::ie_cpu_extension) - add_definitions(-DWITH_EXTENSIONS) -endif() target_link_libraries(${TARGET_NAME} ${InferenceEngine_LIBRARIES} gflags ${OpenCV_LIBRARIES}) -if(TARGET IE::ie_cpu_extension) - target_link_libraries(${TARGET_NAME} IE::ie_cpu_extension) -endif() if(UNIX) target_link_libraries( ${TARGET_NAME} pthread) diff --git a/demos/multichannel_demo/common/decoder.cpp b/demos/multi_channel/common/decoder.cpp similarity index 100% rename from demos/multichannel_demo/common/decoder.cpp rename to demos/multi_channel/common/decoder.cpp diff --git a/demos/multichannel_demo/common/decoder.hpp b/demos/multi_channel/common/decoder.hpp similarity index 98% rename from demos/multichannel_demo/common/decoder.hpp rename to demos/multi_channel/common/decoder.hpp index af58f7e6b7f..feb7a9e81de 100644 --- a/demos/multichannel_demo/common/decoder.hpp +++ b/demos/multi_channel/common/decoder.hpp @@ -34,6 +34,7 @@ class Decoder final { explicit Decoder(const Settings& s); Decoder(const Decoder&) = delete; + Decoder& operator =(const Decoder&) = delete; ~Decoder(); struct Stats { diff --git a/demos/multichannel_demo/common/graph.cpp b/demos/multi_channel/common/graph.cpp similarity index 90% rename from demos/multichannel_demo/common/graph.cpp rename to demos/multi_channel/common/graph.cpp index 5756f04cbd3..3e50233c396 100644 --- a/demos/multichannel_demo/common/graph.cpp +++ b/demos/multi_channel/common/graph.cpp @@ -38,19 +38,9 @@ void loadImgToIEGraph(const cv::Mat& img, size_t batch, void* ieBuffer) { } // namespace void IEGraph::initNetwork(const std::string& deviceName) { - InferenceEngine::CNNNetReader netReader; - - netReader.ReadNetwork(modelPath); - netReader.ReadWeights(weightsPath); - - if (!netReader.isParseSuccess()) { - throw std::logic_error("Failed to parse model!"); - } + auto cnnNetwork = ie.ReadNetwork(modelPath); if (deviceName.find("CPU") != std::string::npos) { -#ifdef WITH_EXTENSIONS - ie.AddExtension(std::make_shared(), "CPU"); -#endif ie.SetConfig({{InferenceEngine::PluginConfigParams::KEY_CPU_BIND_THREAD, "NO"}}, "CPU"); } if (!cpuExtensionPath.empty()) { @@ -67,26 +57,26 @@ void IEGraph::initNetwork(const std::string& deviceName) { // Set batch size if (batchSize > 1) { - auto inShapes = netReader.getNetwork().getInputShapes(); + auto inShapes = cnnNetwork.getInputShapes(); for (auto& pair : inShapes) { auto& dims = pair.second; if (!dims.empty()) { dims[0] = batchSize; } } - netReader.getNetwork().reshape(inShapes); + cnnNetwork.reshape(inShapes); } InferenceEngine::ExecutableNetwork network; - network = ie.LoadNetwork(netReader.getNetwork(), deviceName); + network = ie.LoadNetwork(cnnNetwork, deviceName); - InferenceEngine::InputsDataMap inputInfo(netReader.getNetwork().getInputsInfo()); + InferenceEngine::InputsDataMap inputInfo(cnnNetwork.getInputsInfo()); if (inputInfo.size() != 1) { throw std::logic_error("Face Detection network should have only one input"); } inputDataBlobName = inputInfo.begin()->first; - InferenceEngine::OutputsDataMap outputInfo(netReader.getNetwork().getOutputsInfo()); + InferenceEngine::OutputsDataMap outputInfo(cnnNetwork.getOutputsInfo()); outputDataBlobNames.reserve(outputInfo.size()); for (const auto& i : outputInfo) { outputDataBlobNames.push_back(i.first); @@ -97,6 +87,9 @@ void IEGraph::initNetwork(const std::string& deviceName) { availableRequests.push(req); } + if (postLoad != nullptr) + postLoad(outputDataBlobNames, cnnNetwork); + availableRequests.front()->StartAsync(); availableRequests.front()->Wait(InferenceEngine::IInferRequest::WaitMode::RESULT_READY); } @@ -119,9 +112,8 @@ void IEGraph::start(GetterFunc getterFunc, PostprocessingFunc postprocessingFunc vframes.push_back(std::make_shared(vframe)); ++b; } else { - if (terminate) { - break; - } + terminate = true; + break; } } @@ -188,6 +180,7 @@ void IEGraph::start(GetterFunc getterFunc, PostprocessingFunc postprocessingFunc } condVarBusyRequests.notify_one(); } + condVarBusyRequests.notify_one(); // notify that there will be no new InferRequests }); } @@ -195,15 +188,21 @@ IEGraph::IEGraph(const InitParams& p): perfTimerPreprocess(p.collectStats ? PerfTimer::DefaultIterationsCount : 0), perfTimerInfer(p.collectStats ? PerfTimer::DefaultIterationsCount : 0), confidenceThreshold(0.5f), batchSize(p.batchSize), - modelPath(p.modelPath), weightsPath(p.weightsPath), + modelPath(p.modelPath), cpuExtensionPath(p.cpuExtPath), cldnnConfigPath(p.cldnnConfigPath), printPerfReport(p.reportPerf), deviceName(p.deviceName), maxRequests(p.maxRequests) { assert(p.maxRequests > 0); + postLoad = p.postLoadFunc; initNetwork(p.deviceName); } +bool IEGraph::isRunning() { + std::lock_guard lock(mtxBusyRequests); + return !terminate || !busyBatchRequests.empty(); +} + InferenceEngine::SizeVector IEGraph::getInputDims() const { assert(!availableRequests.empty()); auto inputBlob = availableRequests.front()->GetBlob(inputDataBlobName); @@ -217,8 +216,12 @@ std::vector > IEGraph::getBatchData(cv::Size frameSi { std::unique_lock lock(mtxBusyRequests); condVarBusyRequests.wait(lock, [&]() { - return !busyBatchRequests.empty(); + // wait until the pipeline is stopped or there are new InferRequests + return terminate || !busyBatchRequests.empty(); }); + if (busyBatchRequests.empty()) { + return {}; // woke up because of termination, so leave if nothing to preces + } vframes = std::move(busyBatchRequests.front().vfPtrVec); req = std::move(busyBatchRequests.front().req); startTime = std::move(busyBatchRequests.front().startTime); diff --git a/demos/multichannel_demo/common/graph.hpp b/demos/multi_channel/common/graph.hpp similarity index 93% rename from demos/multichannel_demo/common/graph.hpp rename to demos/multi_channel/common/graph.hpp index 300c0e650aa..261c9e96785 100644 --- a/demos/multichannel_demo/common/graph.hpp +++ b/demos/multi_channel/common/graph.hpp @@ -45,7 +45,6 @@ class IEGraph{ std::size_t batchSize; std::string modelPath; - std::string weightsPath; std::string cpuExtensionPath; std::string cldnnConfigPath; @@ -77,6 +76,8 @@ class IEGraph{ GetterFunc getter; using PostprocessingFunc = std::function(InferenceEngine::InferRequest::Ptr, const std::vector&, cv::Size)>; PostprocessingFunc postprocessing; + using PostLoadFunc = std::function&, InferenceEngine::CNNNetwork&)>; + PostLoadFunc postLoad; std::thread getterThread; void initNetwork(const std::string& deviceName); @@ -88,16 +89,18 @@ class IEGraph{ bool collectStats = false; bool reportPerf = false; std::string modelPath; - std::string weightsPath; std::string cpuExtPath; std::string cldnnConfigPath; std::string deviceName; + PostLoadFunc postLoadFunc = nullptr; }; explicit IEGraph(const InitParams& p); void start(GetterFunc getterFunc, PostprocessingFunc postprocessingFunc); + bool isRunning(); + InferenceEngine::SizeVector getInputDims() const; std::vector> getBatchData(cv::Size windowSize); diff --git a/demos/multichannel_demo/common/input.cpp b/demos/multi_channel/common/input.cpp similarity index 86% rename from demos/multichannel_demo/common/input.cpp rename to demos/multi_channel/common/input.cpp index 14b25e7153d..048139a0a2e 100644 --- a/demos/multichannel_demo/common/input.cpp +++ b/demos/multi_channel/common/input.cpp @@ -34,7 +34,7 @@ class VideoSource { public: - virtual bool init() = 0; + virtual bool isRunning() const = 0; virtual void start() = 0; @@ -142,7 +142,7 @@ class VideoSourceStreamFile : public VideoSource { VideoStream stream; - std::atomic_bool terminate = {false}; + std::atomic_bool running = {false}; std::atomic_bool is_decoding = {false}; std::mutex mutex; @@ -170,12 +170,14 @@ class VideoSourceStreamFile : public VideoSource { queueSize(queueSize_), perfTimer(collectStats_ ? PerfTimer::DefaultIterationsCount : 0) { } - bool init() { return true; } + bool isRunning() const override { + return running; + } void start() { - terminate = false; + running = true; workThread = std::thread([&]() { - while (!terminate) { + while (running) { { cv::Mat frame; { @@ -203,7 +205,7 @@ class VideoSourceStreamFile : public VideoSource { std::unique_lock lock(mutex); condVar.wait(lock, [&]() { - return !is_decoding && (frameQueue.size() < queueSize || terminate); + return !is_decoding && (frameQueue.size() < queueSize || !running); }); } hasFrame.notify_one(); @@ -212,7 +214,7 @@ class VideoSourceStreamFile : public VideoSource { } void stop() { - terminate = true; + running = false; condVar.notify_one(); if (workThread.joinable()) { workThread.join(); @@ -222,13 +224,13 @@ class VideoSourceStreamFile : public VideoSource { bool read(VideoFrame& frame) { queue_elem_t elem; - if (terminate) + if (!running) return false; { std::unique_lock lock(mutex); hasFrame.wait(lock, [&]() { - return !frameQueue.empty() || terminate; + return !frameQueue.empty() || !running; }); elem = std::move(frameQueue.front()); frameQueue.pop(); @@ -236,7 +238,7 @@ class VideoSourceStreamFile : public VideoSource { condVar.notify_one(); frame.frame = std::move(elem.second); - return elem.first && !terminate; + return elem.first && running; } float getAvgReadTime() const { @@ -249,8 +251,8 @@ class VideoSourceStreamFile : public VideoSource { class VideoSourceOCV : public VideoSource { PerfTimer perfTimer; std::thread workThread; - const bool isAsync = false; - std::atomic_bool terminate = {false}; + const bool isAsync; + std::atomic_bool running = {true}; std::string videoName; std::mutex mutex; @@ -268,9 +270,6 @@ class VideoSourceOCV : public VideoSource { template bool readFrame(cv::Mat& frame); - template - bool readFrameImpl(cv::Mat& frame); - template void startImpl(); @@ -282,7 +281,7 @@ class VideoSourceOCV : public VideoSource { void start(); - bool init(); + bool isRunning() const override; void stop(); @@ -331,7 +330,7 @@ class VideoSourceNative : public VideoSource { void start(); - bool init(); + bool isRunning() const override; bool read(VideoFrame& frame); @@ -364,8 +363,7 @@ void VideoSourceNative::start() { // nothing } -bool VideoSourceNative::init() { - // nothing +bool VideoSourceNative::isRunning() const override { return true; } @@ -440,38 +438,8 @@ bool isNumeric(const std::string& str) { } } // namespace -bool VideoSourceOCV::init() { - static std::mutex initMutex; // HACK: opencv camera init is not thread-safe - std::unique_lock lock(initMutex); - bool res = false; - if (isNumeric(videoName)) { -#ifdef __linux__ - res = source.open("/dev/video" + videoName); -#else - res = source.open(std::stoi(videoName)); -#endif - } else { - res = source.open(videoName); - } - if (res) { - source.set(cv::CAP_PROP_FOURCC, cv::VideoWriter::fourcc('M', 'J', 'P', 'G')); - } - return res; -} - template bool VideoSourceOCV::readFrame(cv::Mat& frame) { - if (!source.isOpened() && !init()) { - return false; - } - if (!readFrameImpl(frame)) { - return init() && readFrameImpl(frame); - } - return true; -} - -template -bool VideoSourceOCV::readFrameImpl(cv::Mat& frame) { if (CollectStats) { ScopedTimer st(perfTimer); return source.read(frame); @@ -483,38 +451,40 @@ bool VideoSourceOCV::readFrameImpl(cv::Mat& frame) { VideoSourceOCV::VideoSourceOCV(bool async, bool collectStats_, const std::string& name, size_t queueSize_, size_t pollingTimeMSec_, bool realFps_): - perfTimer(collectStats_ ? PerfTimer::DefaultIterationsCount : 0), - isAsync(async), videoName(name), - realFps(realFps_), - queueSize(queueSize_), - pollingTimeMSec(pollingTimeMSec_) {} + perfTimer(collectStats_ ? PerfTimer::DefaultIterationsCount : 0), + isAsync(async), videoName(name), + realFps(realFps_), + queueSize(queueSize_), + pollingTimeMSec(pollingTimeMSec_) { + if (isNumeric(videoName)) { + if (!source.open(std::stoi(videoName))) { + throw std::runtime_error("Can't open " + videoName + " with cv::VideoCapture::open(int)"); + } + } else { + if (!source.open(videoName)) { + throw std::runtime_error("Can't open " + videoName + " with cv::VideoCapture::open(std::string)"); + } + } + source.set(cv::CAP_PROP_FOURCC, cv::VideoWriter::fourcc('M', 'J', 'P', 'G')); +} VideoSourceOCV::~VideoSourceOCV() { stop(); } +bool VideoSourceOCV::isRunning() const { + return running; +} + template void VideoSourceOCV::thread_fn(VideoSourceOCV *vs) { - while (!vs->terminate) { + while (vs->running) { cv::Mat frame; - bool result = false; - while (!((result = vs->readFrame(frame)) || vs->terminate)) { - std::unique_lock lock(vs->mutex); - if (vs->queue.empty() || vs->queue.back().first) { - vs->queue.push({false, frame}); - lock.unlock(); - vs->hasFrame.notify_one(); - lock.lock(); - } - std::chrono::milliseconds timeout(vs->pollingTimeMSec); - vs->condVar.wait_for(lock, - timeout, - [&]() { - return vs->terminate.load(); - }); + const bool result = vs->readFrame(frame); + if (!result) { + vs->running = false; // stop() also affects running, so override it only when out of frames } - - if (vs->queue.size() < vs->queueSize) { + if (vs->queue.size() < vs->queueSize || !result) { // queue has space or source run out of frames std::unique_lock lock(vs->mutex); vs->queue.push({result, frame}); } @@ -525,7 +495,7 @@ void VideoSourceOCV::thread_fn(VideoSourceOCV *vs) { template void VideoSourceOCV::startImpl() { if (isAsync) { - terminate = false; + running = true; workThread = std::thread(&VideoSourceOCV::thread_fn, this); } } @@ -540,7 +510,7 @@ void VideoSourceOCV::start() { void VideoSourceOCV::stop() { if (isAsync) { - terminate = true; + running = false; condVar.notify_one(); if (workThread.joinable()) { workThread.join(); @@ -550,20 +520,17 @@ void VideoSourceOCV::stop() { bool VideoSourceOCV::read(cv::Mat& frame) { if (isAsync) { - size_t count = 0; - bool res = false; + bool res; { std::unique_lock lock(mutex); hasFrame.wait(lock, [&]() { - return !queue.empty() || terminate; + return !queue.empty() || !running; }); res = queue.front().first; frame = queue.front().second; if (realFps || queue.size() > 1 || queueSize == 1) { queue.pop(); } - count = queue.size(); - (void)count; } condVar.notify_one(); return res; @@ -608,6 +575,13 @@ VideoSources::~VideoSources() { // nothing } +bool VideoSources::isRunning() const { + // when one of VideoSources will be out of frames, it will stop IEGraph, + // so this isRunning() requires that all inputs were running + return std::all_of(inputs.begin(), inputs.end(), + [](const std::unique_ptr& input){return input->isRunning();}); +} + void VideoSources::openVideo(const std::string& source, bool native) { #ifdef USE_NATIVE_CAMERA_API if (native) { @@ -626,10 +600,10 @@ void VideoSources::openVideo(const std::string& source, bool native) { std::unique_ptr newSrc(new VideoSourceNative(*this, controller, dev, camSettings, queueSize, realFps, collectStats)); inputs.emplace_back(std::move(newSrc)); + } else { #else - if (false) { + { #endif - } else { #if defined(USE_LIBVA) const std::string extension = ".mjpeg"; std::unique_ptr newSrc; @@ -643,11 +617,7 @@ void VideoSources::openVideo(const std::string& source, bool native) { std::unique_ptr newSrc(new VideoSourceOCV(isAsync, collectStats, source, queueSize, pollingTimeMSec, realFps)); #endif - if (newSrc->init()) { - inputs.emplace_back(std::move(newSrc)); - } else { - throw std::runtime_error("Cannot open cv::VideoCapture"); - } + inputs.emplace_back(std::move(newSrc)); } } diff --git a/demos/multichannel_demo/common/input.hpp b/demos/multi_channel/common/input.hpp similarity index 92% rename from demos/multichannel_demo/common/input.hpp rename to demos/multi_channel/common/input.hpp index 413ee022b26..779cc825e05 100644 --- a/demos/multichannel_demo/common/input.hpp +++ b/demos/multi_channel/common/input.hpp @@ -40,7 +40,7 @@ class VideoFrame final { Detections detections; VideoFrame() = default; - void operator =(VideoFrame const& vf) = delete; + VideoFrame& operator =(VideoFrame const& vf) = delete; }; class VideoSource; @@ -58,8 +58,8 @@ class VideoSources { std::mutex decode_mutex; // hardware decoding enqueue lock std::vector> inputs; - const bool isAsync = false; - const bool collectStats = false; + const bool isAsync; + const bool collectStats; bool realFps; @@ -90,6 +90,8 @@ class VideoSources { void start(); + virtual bool isRunning() const; + bool getFrame(size_t index, VideoFrame& frame); struct Stats { diff --git a/demos/multichannel_demo/common/multicam/CMakeLists.txt b/demos/multi_channel/common/multicam/CMakeLists.txt similarity index 100% rename from demos/multichannel_demo/common/multicam/CMakeLists.txt rename to demos/multi_channel/common/multicam/CMakeLists.txt diff --git a/demos/multichannel_demo/common/multicam/camera.cpp b/demos/multi_channel/common/multicam/camera.cpp similarity index 100% rename from demos/multichannel_demo/common/multicam/camera.cpp rename to demos/multi_channel/common/multicam/camera.cpp diff --git a/demos/multichannel_demo/common/multicam/camera.hpp b/demos/multi_channel/common/multicam/camera.hpp similarity index 100% rename from demos/multichannel_demo/common/multicam/camera.hpp rename to demos/multi_channel/common/multicam/camera.hpp diff --git a/demos/multichannel_demo/common/multicam/controller.cpp b/demos/multi_channel/common/multicam/controller.cpp similarity index 100% rename from demos/multichannel_demo/common/multicam/controller.cpp rename to demos/multi_channel/common/multicam/controller.cpp diff --git a/demos/multichannel_demo/common/multicam/controller.hpp b/demos/multi_channel/common/multicam/controller.hpp similarity index 100% rename from demos/multichannel_demo/common/multicam/controller.hpp rename to demos/multi_channel/common/multicam/controller.hpp diff --git a/demos/multichannel_demo/common/multicam/utils.cpp b/demos/multi_channel/common/multicam/utils.cpp similarity index 100% rename from demos/multichannel_demo/common/multicam/utils.cpp rename to demos/multi_channel/common/multicam/utils.cpp diff --git a/demos/multichannel_demo/common/multicam/utils.hpp b/demos/multi_channel/common/multicam/utils.hpp similarity index 100% rename from demos/multichannel_demo/common/multicam/utils.hpp rename to demos/multi_channel/common/multicam/utils.hpp diff --git a/demos/multi_channel/common/multichannel_params.hpp b/demos/multi_channel/common/multichannel_params.hpp new file mode 100644 index 00000000000..471e3bff9eb --- /dev/null +++ b/demos/multi_channel/common/multichannel_params.hpp @@ -0,0 +1,51 @@ +// Copyright (C) 2018-2019 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include + +static const char help_message[] = "Print a usage message"; +static const char model_path_message[] = "Required. Path to an .xml file with a trained model."; +static const char target_device_message[] = "Optional. Specify the target device for a network (the list of available devices is shown below). " + "Default value is CPU. Use \"-d HETERO:\" format to specify HETERO plugin. " + "The demo looks for a suitable plugin for a specified device."; +static const char performance_counter_message[] = "Optional. Enable per-layer performance report"; +static const char custom_cldnn_message[] = "Required for GPU custom kernels. " + "Absolute path to an .xml file with the kernels descriptions"; +static const char custom_cpu_library_message[] = "Required for CPU custom layers. " + "Absolute path to a shared library with the kernels implementations"; +static const char no_show_processed_video[] = "Optional. Do not show processed video."; +static const char num_cameras[] = "Optional. Maximum number of processed camera inputs (web cameras)"; +static const char batch_size[] = "Optional. Batch size for processing (the number of frames processed per infer request)"; +static const char num_infer_requests[] = "Optional. Number of infer requests"; +static const char input_queue_size[] = "Optional. Frame queue size for input channels"; +static const char fps_sampling_period[] = "Optional. FPS measurement sampling period between timepoints in msec"; +static const char num_sampling_periods[] = "Optional. Number of sampling periods"; +static const char show_statistics[] = "Optional. Enable statistics report"; +static const char duplication_channel_number[] = "Optional. Enable and specify the number of channels additionally copied from real sources"; +static const char real_input_fps[] = "Optional. Disable input frames caching, for maximum throughput pipeline"; +static const char input_video[] = "Optional. Specify full path to input video files"; +static const char utilization_monitors_message[] = "Optional. List of monitors to show initially."; + +DEFINE_bool(h, false, help_message); +DEFINE_string(m, "", model_path_message); +DEFINE_string(d, "CPU", target_device_message); +DEFINE_bool(pc, false, performance_counter_message); +DEFINE_string(c, "", custom_cldnn_message); +DEFINE_string(l, "", custom_cpu_library_message); +DEFINE_bool(no_show, false, no_show_processed_video); +DEFINE_uint32(nc, 0, num_cameras); +DEFINE_uint32(bs, 1, batch_size); +DEFINE_uint32(nireq, 5, num_infer_requests); +DEFINE_uint32(n_iqs, 5, input_queue_size); +DEFINE_uint32(fps_sp, 1000, fps_sampling_period); +DEFINE_uint32(n_sp, 10, num_sampling_periods); +DEFINE_bool(show_stats, false, show_statistics); +DEFINE_uint32(duplicate_num, 0, duplication_channel_number); +DEFINE_bool(real_input_fps, false, real_input_fps); +DEFINE_string(i, "", input_video); +DEFINE_string(u, "", utilization_monitors_message); diff --git a/demos/multichannel_demo/common/output.cpp b/demos/multi_channel/common/output.cpp similarity index 98% rename from demos/multichannel_demo/common/output.cpp rename to demos/multi_channel/common/output.cpp index 4aec97fdf6e..3e0c1e8da12 100644 --- a/demos/multichannel_demo/common/output.cpp +++ b/demos/multi_channel/common/output.cpp @@ -40,7 +40,7 @@ void AsyncOutput::start() { condVar.wait(lock, [&]() { return !queue.empty() || terminate; }); - if (terminate) { + if (queue.empty()) { break; } @@ -62,7 +62,6 @@ void AsyncOutput::start() { }); } - bool AsyncOutput::isAlive() const { return !terminate; } diff --git a/demos/multichannel_demo/common/output.hpp b/demos/multi_channel/common/output.hpp similarity index 100% rename from demos/multichannel_demo/common/output.hpp rename to demos/multi_channel/common/output.hpp diff --git a/demos/multichannel_demo/common/perf_timer.cpp b/demos/multi_channel/common/perf_timer.cpp similarity index 100% rename from demos/multichannel_demo/common/perf_timer.cpp rename to demos/multi_channel/common/perf_timer.cpp diff --git a/demos/multichannel_demo/common/perf_timer.hpp b/demos/multi_channel/common/perf_timer.hpp similarity index 100% rename from demos/multichannel_demo/common/perf_timer.hpp rename to demos/multi_channel/common/perf_timer.hpp diff --git a/demos/multichannel_demo/common/threading.cpp b/demos/multi_channel/common/threading.cpp similarity index 100% rename from demos/multichannel_demo/common/threading.cpp rename to demos/multi_channel/common/threading.cpp diff --git a/demos/multichannel_demo/common/threading.hpp b/demos/multi_channel/common/threading.hpp similarity index 100% rename from demos/multichannel_demo/common/threading.hpp rename to demos/multi_channel/common/threading.hpp diff --git a/demos/multichannel_demo/fd/CMakeLists.txt b/demos/multi_channel/face_detection_demo/CMakeLists.txt similarity index 90% rename from demos/multichannel_demo/fd/CMakeLists.txt rename to demos/multi_channel/face_detection_demo/CMakeLists.txt index fac14a84a1d..5b497fcf401 100644 --- a/demos/multichannel_demo/fd/CMakeLists.txt +++ b/demos/multi_channel/face_detection_demo/CMakeLists.txt @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -set(TARGET_NAME "multi-channel-face-detection-demo") +set(TARGET_NAME "multi_channel_face_detection_demo") if( BUILD_DEMO_NAME AND NOT ${BUILD_DEMO_NAME} STREQUAL ${TARGET_NAME} ) message(STATUS "DEMO ${TARGET_NAME} SKIPPED") @@ -60,16 +60,8 @@ if(MULTICHANNEL_DEMO_USE_TBB) endif() endif() -if(TARGET IE::ie_cpu_extension) - add_definitions(-DWITH_EXTENSIONS) -endif() - target_link_libraries(${TARGET_NAME} ${InferenceEngine_LIBRARIES} gflags ${OpenCV_LIBRARIES} common) -if(TARGET IE::ie_cpu_extension) - target_link_libraries(${TARGET_NAME} IE::ie_cpu_extension) -endif() - if(UNIX) target_link_libraries( ${TARGET_NAME} pthread) endif() @@ -83,3 +75,5 @@ if(NOT TARGET ie_samples) endif() add_dependencies(ie_samples ${TARGET_NAME}) + +target_link_libraries(${TARGET_NAME} monitors) diff --git a/demos/multichannel_demo/fd/README.md b/demos/multi_channel/face_detection_demo/README.md similarity index 93% rename from demos/multichannel_demo/fd/README.md rename to demos/multi_channel/face_detection_demo/README.md index cb3d6a005b5..85c38892f36 100644 --- a/demos/multichannel_demo/fd/README.md +++ b/demos/multi_channel/face_detection_demo/README.md @@ -22,10 +22,10 @@ On the start-up, the application reads command line parameters and loads the spe ## Running Running the application with the `-h` option yields the following usage message: -```sh -./multi-channel-face-detection-demo -h +``` +./multi_channel_face_detection_demo -h -multi-channel-face-detection-demo [OPTION] +multi_channel_face_detection_demo [OPTION] Options: -h Print a usage message @@ -47,7 +47,7 @@ Options: -duplicate_num Optional. Enable and specify the number of channels additionally copied from real sources -real_input_fps Optional. Disable input frames caching for maximum throughput pipeline -i Optional. Specify full path to input video files - + -u Optional. List of monitors to show initially. ``` To run the demo, you can use public or pre-trained models. To download the pre-trained models, use the OpenVINO [Model Downloader](../../../tools/downloader/README.md) or go to [https://download.01.org/opencv/](https://download.01.org/opencv/). @@ -56,14 +56,12 @@ To run the demo, you can use public or pre-trained models. To download the pre-t For example, to run the demo with the pre-trained face detection model on FPGA with fallback on CPU, with one single camera, use the following command: ```sh -./multi-channel-face-detection-demo -m face-detection-retail-0004.xml --l /intel64/Release/lib/libcpu_extension.so -d HETERO:FPGA,CPU -nc 1 +./multi_channel_face_detection_demo -m face-detection-retail-0004.xml -d HETERO:FPGA,CPU -nc 1 ``` To run the demo using two recorded video files, use the following command: ```sh -./multi-channel-face-detection-demo -m face-detection-retail-0004.xml --l /intel64/Release/lib/libcpu_extension.so -d HETERO:FPGA,CPU -i /path/to/file1 /path/to/file2 +./multi_channel_face_detection_demo -m face-detection-retail-0004.xml -d HETERO:FPGA,CPU -i /path/to/file1 /path/to/file2 ``` Video files will be processed repeatedly. @@ -87,7 +85,7 @@ General parameter for input video source is `-i`. Use it to specify video files To see all available web cameras, run the `ls /dev/video*` command. You will get output similar to the following: -```sh +``` user@user-PC:~ $ ls /dev/video* /dev/video0 /dev/video1 /dev/video2 ``` @@ -102,7 +100,7 @@ Alternatively, you can just set `-nc 3`, which simplifies application usage. If your cameras are connected to PC with indexes gap (for example, `0,1,3`), use the `-i` parameter. -IP сameras support: +IP cameras support: ``` -i rtsp://camera_address_1/ rtsp://camera_address_2/ ``` diff --git a/demos/multichannel_demo/fd/main.cpp b/demos/multi_channel/face_detection_demo/main.cpp similarity index 92% rename from demos/multichannel_demo/fd/main.cpp rename to demos/multi_channel/face_detection_demo/main.cpp index 7f2ebd8fc4f..2996cfbe4a3 100644 --- a/demos/multichannel_demo/fd/main.cpp +++ b/demos/multi_channel/face_detection_demo/main.cpp @@ -27,8 +27,8 @@ #include +#include #include - #include #include "input.hpp" @@ -45,11 +45,11 @@ namespace { */ void showUsage() { std::cout << std::endl; - std::cout << "multichannel_face_detection [OPTION]" << std::endl; + std::cout << "multi_channel_face_detection_demo [OPTION]" << std::endl; std::cout << "Options:" << std::endl; std::cout << std::endl; std::cout << " -h " << help_message << std::endl; - std::cout << " -m \"\" " << face_detection_model_message<< std::endl; + std::cout << " -m \"\" " << model_path_message<< std::endl; std::cout << " -l \"\" " << custom_cpu_library_message << std::endl; std::cout << " Or" << std::endl; std::cout << " -c \"\" " << custom_cldnn_message << std::endl; @@ -67,6 +67,7 @@ void showUsage() { std::cout << " -duplicate_num " << duplication_channel_number << std::endl; std::cout << " -real_input_fps " << real_input_fps << std::endl; std::cout << " -i " << input_video << std::endl; + std::cout << " -u " << utilization_monitors_message << std::endl; } bool ParseAndCheckCommandLine(int argc, char *argv[]) { @@ -109,7 +110,7 @@ struct Face { Face(cv::Rect2f r, float c, unsigned char a, unsigned char g): rect(r), confidence(c), age(a), gender(g) {} }; -void drawDetections(cv::Mat& img, const std::vector detections) { +void drawDetections(cv::Mat& img, const std::vector& detections) { for (const Face& f : detections) { cv::Rect ri(static_cast(f.rect.x*img.cols), static_cast(f.rect.y*img.rows), static_cast(f.rect.width*img.cols), static_cast(f.rect.height*img.rows)); @@ -151,7 +152,8 @@ DisplayParams prepareDisplayParams(size_t count) { void displayNSources(const std::vector>& data, float time, const std::string& stats, - DisplayParams params) { + DisplayParams params, + Presenter& presenter) { cv::Mat windowImage = cv::Mat::zeros(params.windowSize, CV_8UC3); auto loopBody = [&](size_t i) { auto& elem = data[i]; @@ -192,6 +194,7 @@ void displayNSources(const std::vector>& data, loopBody(i); } #endif + presenter.drawGraphs(windowImage); drawStats(); char str[256]; @@ -214,7 +217,6 @@ int main(int argc, char* argv[]) { return 0; } - std::string weightsPath; std::string modelPath = FLAGS_m; std::size_t found = modelPath.find_last_of("."); if (found > modelPath.size()) { @@ -222,9 +224,7 @@ int main(int argc, char* argv[]) { slog::info << "Expected to be .xml" << slog::endl; return -1; } - weightsPath = modelPath.substr(0, found) + ".bin"; slog::info << "Model path: " << modelPath << slog::endl; - slog::info << "Weights path: " << weightsPath << slog::endl; IEGraph::InitParams graphParams; graphParams.batchSize = FLAGS_bs; @@ -232,7 +232,6 @@ int main(int argc, char* argv[]) { graphParams.collectStats = FLAGS_show_stats; graphParams.reportPerf = FLAGS_pc; graphParams.modelPath = modelPath; - graphParams.weightsPath = weightsPath; graphParams.cpuExtPath = FLAGS_l; graphParams.cldnnConfigPath = FLAGS_c; graphParams.deviceName = FLAGS_d; @@ -346,6 +345,9 @@ int main(int argc, char* argv[]) { } std::cout << std::endl; + cv::Size graphSize{static_cast(params.windowSize.width / 4), 60}; + Presenter presenter(FLAGS_u, params.windowSize.height - graphSize.height - 10, graphSize); + const size_t outputQueueSize = 1; AsyncOutput output(FLAGS_show_stats, outputQueueSize, [&](const std::vector>& result) { @@ -354,9 +356,11 @@ int main(int argc, char* argv[]) { std::unique_lock lock(statMutex); str = statStream.str(); } - displayNSources(result, averageFps, str, params); + displayNSources(result, averageFps, str, params, presenter); + int key = cv::waitKey(1); + presenter.handleKey(key); - return (cv::waitKey(1) != 27); + return (key != 27); }); output.start(); @@ -370,11 +374,16 @@ int main(int argc, char* argv[]) { size_t perfItersCounter = 0; - while (true) { + while (sources.isRunning() || network->isRunning()) { bool readData = true; while (readData) { auto br = network->getBatchData(params.frameSize); + if (br.empty()) { + break; // IEGraph::getBatchData had nothing to process and returned. That means it was stopped + } for (size_t i = 0; i < br.size(); i++) { + // this approach waits for the next input image for sourceIdx. If provided a single image, + // it may not show results, especially if -real_input_fps is enabled auto val = static_cast(br[i]->sourceIdx); auto it = find_if(batchRes.begin(), batchRes.end(), [val] (const std::shared_ptr& vf) { return vf->sourceIdx == val; } ); if (it != batchRes.end()) { @@ -448,6 +457,8 @@ int main(int argc, char* argv[]) { } network.reset(); + + std::cout << presenter.reportMeans() << '\n'; } catch (const std::exception& error) { slog::err << error.what() << slog::endl; diff --git a/demos/multichannel_demo/models.lst b/demos/multi_channel/face_detection_demo/models.lst similarity index 85% rename from demos/multichannel_demo/models.lst rename to demos/multi_channel/face_detection_demo/models.lst index 205180adf86..b08884eca53 100644 --- a/demos/multichannel_demo/models.lst +++ b/demos/multi_channel/face_detection_demo/models.lst @@ -2,4 +2,3 @@ face-detection-adas-???? face-detection-adas-binary-???? face-detection-retail-???? -human-pose-estimation-???? diff --git a/demos/multichannel_demo/fd/multichannel_face_detection_params.hpp b/demos/multi_channel/face_detection_demo/multichannel_face_detection_params.hpp similarity index 64% rename from demos/multichannel_demo/fd/multichannel_face_detection_params.hpp rename to demos/multi_channel/face_detection_demo/multichannel_face_detection_params.hpp index 9de1ad57da9..b4ef8ee7ec2 100644 --- a/demos/multichannel_demo/fd/multichannel_face_detection_params.hpp +++ b/demos/multi_channel/face_detection_demo/multichannel_face_detection_params.hpp @@ -8,9 +8,6 @@ #include #include -/// @brief message for probability threshold argument static const char thresh_output_message[] = "Optional. Probability threshold for detections"; -/// \brief Flag to output raw scoring results
-/// It is an optional parameter. Ignored for human-pose-estimation DEFINE_double(t, 0.5, thresh_output_message); diff --git a/demos/multichannel_demo/hpe/CMakeLists.txt b/demos/multi_channel/human_pose_estimation_demo/CMakeLists.txt similarity index 90% rename from demos/multichannel_demo/hpe/CMakeLists.txt rename to demos/multi_channel/human_pose_estimation_demo/CMakeLists.txt index 4789bcd36b1..7fe48237dda 100644 --- a/demos/multichannel_demo/hpe/CMakeLists.txt +++ b/demos/multi_channel/human_pose_estimation_demo/CMakeLists.txt @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -set(TARGET_NAME "multi-channel-human-pose-estimation-demo") +set(TARGET_NAME "multi_channel_human_pose_estimation_demo") if( BUILD_DEMO_NAME AND NOT ${BUILD_DEMO_NAME} STREQUAL ${TARGET_NAME} ) message(STATUS "DEMO ${TARGET_NAME} SKIPPED") @@ -60,13 +60,7 @@ if(MULTICHANNEL_DEMO_USE_TBB) endif() endif() -if(TARGET IE::ie_cpu_extension) - add_definitions(-DWITH_EXTENSIONS) -endif() target_link_libraries(${TARGET_NAME} ${InferenceEngine_LIBRARIES} gflags ${OpenCV_LIBRARIES} common) -if(TARGET IE::ie_cpu_extension) - target_link_libraries(${TARGET_NAME} IE::ie_cpu_extension) -endif() if(UNIX) target_link_libraries( ${TARGET_NAME} pthread) @@ -81,3 +75,5 @@ if(COMMAND add_cpplint_target) endif() add_dependencies(ie_samples ${TARGET_NAME}) + +target_link_libraries(${TARGET_NAME} monitors) diff --git a/demos/multichannel_demo/hpe/README.md b/demos/multi_channel/human_pose_estimation_demo/README.md similarity index 92% rename from demos/multichannel_demo/hpe/README.md rename to demos/multi_channel/human_pose_estimation_demo/README.md index bec055f1d54..06922caf992 100644 --- a/demos/multichannel_demo/hpe/README.md +++ b/demos/multi_channel/human_pose_estimation_demo/README.md @@ -22,9 +22,9 @@ On the start-up, the application reads command line parameters and loads the spe ## Running Running the application with the `-h` option yields the following usage message: -```sh -./multi-channel-human-pose-estimation-demo -h -multi-channel-human-pose-estimation-demo [OPTION] +``` +./multi_channel_human_pose_estimation_demo -h +multi_channel_human_pose_estimation_demo [OPTION] Options: -h Print a usage message -m "" Required. Path to an .xml file with a trained model. @@ -44,6 +44,7 @@ Options: -duplicate_num Optional. Enable and specify the number of channels additionally copied from real sources -real_input_fps Optional. Disable input frames caching for maximum throughput pipeline -i "" Optional. Specify a full path to input video files + -u Optional. List of monitors to show initially. ``` Running the application with an empty list of options yields the usage message given above and an error message. @@ -54,14 +55,12 @@ To run the demo, you can use public or pre-trained models. To download the pre-t For example, to run the demo with the pre-trained Human Pose Estimation model on FPGA with fallback on CPU with one camera, use the following command: ```sh -./multi-channel-human-pose-estimation-demo -m /human-pose-estimation-0001.xml --l /intel64/Release/lib/libcpu_extension.so -d HETERO:FPGA,CPU -nc 1 +./multi_channel_human_pose_estimation_demo -m /human-pose-estimation-0001.xml -d HETERO:FPGA,CPU -nc 1 ``` To run the demo using two recorded video files, use the following command: ```sh -./multi-channel-human-pose-estimation-demo -m /human-pose-estimation-0001.xml --l /intel64/Release/lib/libcpu_extension.so -d HETERO:FPGA,CPU -i /path/to/file1 /path/to/file2 +./multi_channel_human_pose_estimation_demo -m /human-pose-estimation-0001.xml -d HETERO:FPGA,CPU -i /path/to/file1 /path/to/file2 ``` Video files will be processed repeatedly. @@ -86,7 +85,7 @@ General parameter for input video source is `-i`. Use it to specify video files To see all available web cameras, run the `ls /dev/video*` command. You will get output similar to the following: -```sh +``` user@user-PC:~ $ ls /dev/video* /dev/video0 /dev/video1 /dev/video2 ``` diff --git a/demos/multichannel_demo/hpe/human_pose.cpp b/demos/multi_channel/human_pose_estimation_demo/human_pose.cpp similarity index 100% rename from demos/multichannel_demo/hpe/human_pose.cpp rename to demos/multi_channel/human_pose_estimation_demo/human_pose.cpp diff --git a/demos/multichannel_demo/hpe/human_pose.hpp b/demos/multi_channel/human_pose_estimation_demo/human_pose.hpp similarity index 100% rename from demos/multichannel_demo/hpe/human_pose.hpp rename to demos/multi_channel/human_pose_estimation_demo/human_pose.hpp diff --git a/demos/multichannel_demo/hpe/main.cpp b/demos/multi_channel/human_pose_estimation_demo/main.cpp similarity index 93% rename from demos/multichannel_demo/hpe/main.cpp rename to demos/multi_channel/human_pose_estimation_demo/main.cpp index a6fd3e5457c..b7ebd07fbd2 100644 --- a/demos/multichannel_demo/hpe/main.cpp +++ b/demos/multi_channel/human_pose_estimation_demo/main.cpp @@ -38,8 +38,8 @@ #include +#include #include - #include #include "input.hpp" @@ -61,11 +61,11 @@ namespace { */ void showUsage() { std::cout << std::endl; - std::cout << "multi-channel-human-pose-estimation-demo [OPTION]" << std::endl; + std::cout << "multi_channel_human_pose_estimation_demo [OPTION]" << std::endl; std::cout << "Options:" << std::endl; std::cout << std::endl; std::cout << " -h " << help_message << std::endl; - std::cout << " -m \"\" " << face_detection_model_message<< std::endl; + std::cout << " -m \"\" " << model_path_message<< std::endl; std::cout << " -l \"\" " << custom_cpu_library_message << std::endl; std::cout << " Or" << std::endl; std::cout << " -c \"\" " << custom_cldnn_message << std::endl; @@ -82,6 +82,7 @@ void showUsage() { std::cout << " -duplicate_num " << duplication_channel_number << std::endl; std::cout << " -real_input_fps " << real_input_fps << std::endl; std::cout << " -i " << input_video << std::endl; + std::cout << " -u " << utilization_monitors_message << std::endl; } bool ParseAndCheckCommandLine(int argc, char *argv[]) { @@ -149,7 +150,8 @@ DisplayParams prepareDisplayParams(size_t count) { void displayNSources(const std::vector>& data, float time, const std::string& stats, - DisplayParams params) { + DisplayParams params, + Presenter& presenter) { cv::Mat windowImage = cv::Mat::zeros(params.windowSize, CV_8UC3); auto loopBody = [&](size_t i) { auto& elem = data[i]; @@ -190,6 +192,7 @@ void displayNSources(const std::vector>& data, loopBody(i); } #endif + presenter.drawGraphs(windowImage); drawStats(); char str[256]; @@ -212,7 +215,6 @@ int main(int argc, char* argv[]) { return 0; } - std::string weightsPath; std::string modelPath = FLAGS_m; std::size_t found = modelPath.find_last_of("."); if (found > modelPath.size()) { @@ -220,9 +222,7 @@ int main(int argc, char* argv[]) { slog::info << "Expected to be .xml" << slog::endl; return -1; } - weightsPath = modelPath.substr(0, found) + ".bin"; slog::info << "Model path: " << modelPath << slog::endl; - slog::info << "Weights path: " << weightsPath << slog::endl; IEGraph::InitParams graphParams; graphParams.batchSize = FLAGS_bs; @@ -230,7 +230,6 @@ int main(int argc, char* argv[]) { graphParams.collectStats = FLAGS_show_stats; graphParams.reportPerf = FLAGS_pc; graphParams.modelPath = modelPath; - graphParams.weightsPath = weightsPath; graphParams.cpuExtPath = FLAGS_l; graphParams.cldnnConfigPath = FLAGS_c; graphParams.deviceName = FLAGS_d; @@ -344,6 +343,9 @@ int main(int argc, char* argv[]) { } std::cout << std::endl; + cv::Size graphSize{static_cast(params.windowSize.width / 4), 60}; + Presenter presenter(FLAGS_u, params.windowSize.height - graphSize.height - 10, graphSize); + const size_t outputQueueSize = 1; AsyncOutput output(FLAGS_show_stats, outputQueueSize, [&](const std::vector>& result) { @@ -352,9 +354,11 @@ int main(int argc, char* argv[]) { std::unique_lock lock(statMutex); str = statStream.str(); } - displayNSources(result, averageFps, str, params); + displayNSources(result, averageFps, str, params, presenter); + int key = cv::waitKey(1); + presenter.handleKey(key); - return (cv::waitKey(1) != 27); + return (key != 27); }); output.start(); @@ -368,11 +372,16 @@ int main(int argc, char* argv[]) { size_t perfItersCounter = 0; - while (true) { + while (sources.isRunning() || network->isRunning()) { bool readData = true; while (readData) { auto br = network->getBatchData(params.frameSize); + if (br.empty()) { + break; // IEGraph::getBatchData had nothing to process and returned. That means it was stopped + } for (size_t i = 0; i < br.size(); i++) { + // this approach waits for the next input image for sourceIdx. If provided a single image, + // it may not show results, especially if -real_input_fps is enabled auto val = static_cast(br[i]->sourceIdx); auto it = find_if(batchRes.begin(), batchRes.end(), [val] (const std::shared_ptr& vf) { return vf->sourceIdx == val; } ); if (it != batchRes.end()) { @@ -446,6 +455,8 @@ int main(int argc, char* argv[]) { } network.reset(); + + std::cout << presenter.reportMeans() << '\n'; } catch (const std::exception& error) { slog::err << error.what() << slog::endl; diff --git a/demos/multi_channel/human_pose_estimation_demo/models.lst b/demos/multi_channel/human_pose_estimation_demo/models.lst new file mode 100644 index 00000000000..26d650bc499 --- /dev/null +++ b/demos/multi_channel/human_pose_estimation_demo/models.lst @@ -0,0 +1,2 @@ +# This file can be used with the --list option of the model downloader. +human-pose-estimation-???? diff --git a/demos/multichannel_demo/hpe/peak.cpp b/demos/multi_channel/human_pose_estimation_demo/peak.cpp similarity index 98% rename from demos/multichannel_demo/hpe/peak.cpp rename to demos/multi_channel/human_pose_estimation_demo/peak.cpp index 4ffdc7a7594..51b423c1c2b 100644 --- a/demos/multichannel_demo/hpe/peak.cpp +++ b/demos/multi_channel/human_pose_estimation_demo/peak.cpp @@ -18,6 +18,8 @@ #include #include +#include + #include "peak.hpp" Peak::Peak(const int id, const cv::Point2f& pos, const float score) @@ -123,11 +125,11 @@ std::vector groupPeaksToPoses(const std::vector >& const float foundMidPointsRatioThreshold, const int minJointsNumber, const float minSubsetScore) { - const std::vector > limbIdsHeatmap = { + static const std::pair limbIdsHeatmap[] = { {2, 3}, {2, 6}, {3, 4}, {4, 5}, {6, 7}, {7, 8}, {2, 9}, {9, 10}, {10, 11}, {2, 12}, {12, 13}, {13, 14}, {2, 1}, {1, 15}, {15, 17}, {1, 16}, {16, 18}, {3, 17}, {6, 18} }; - const std::vector > limbIdsPaf = { + static const std::pair limbIdsPaf[] = { {31, 32}, {39, 40}, {33, 34}, {35, 36}, {41, 42}, {43, 44}, {19, 20}, {21, 22}, {23, 24}, {25, 26}, {27, 28}, {29, 30}, {47, 48}, {49, 50}, {53, 54}, {51, 52}, {55, 56}, {37, 38}, {45, 46} }; @@ -137,7 +139,7 @@ std::vector groupPeaksToPoses(const std::vector >& candidates.insert(candidates.end(), peaks.begin(), peaks.end()); } std::vector subset(0, HumanPoseByPeaksIndices(keypointsNumber)); - for (size_t k = 0; k < limbIdsPaf.size(); k++) { + for (size_t k = 0; k < arraySize(limbIdsPaf); k++) { std::vector connections; const int mapIdxOffset = keypointsNumber + 1; std::pair scoreMid = { pafs[limbIdsPaf[k].first - mapIdxOffset], diff --git a/demos/multichannel_demo/hpe/peak.hpp b/demos/multi_channel/human_pose_estimation_demo/peak.hpp similarity index 100% rename from demos/multichannel_demo/hpe/peak.hpp rename to demos/multi_channel/human_pose_estimation_demo/peak.hpp diff --git a/demos/multichannel_demo/hpe/postprocess.cpp b/demos/multi_channel/human_pose_estimation_demo/postprocess.cpp similarity index 100% rename from demos/multichannel_demo/hpe/postprocess.cpp rename to demos/multi_channel/human_pose_estimation_demo/postprocess.cpp diff --git a/demos/multichannel_demo/hpe/postprocess.hpp b/demos/multi_channel/human_pose_estimation_demo/postprocess.hpp similarity index 100% rename from demos/multichannel_demo/hpe/postprocess.hpp rename to demos/multi_channel/human_pose_estimation_demo/postprocess.hpp diff --git a/demos/multichannel_demo/hpe/postprocessor.cpp b/demos/multi_channel/human_pose_estimation_demo/postprocessor.cpp similarity index 100% rename from demos/multichannel_demo/hpe/postprocessor.cpp rename to demos/multi_channel/human_pose_estimation_demo/postprocessor.cpp diff --git a/demos/multichannel_demo/hpe/postprocessor.hpp b/demos/multi_channel/human_pose_estimation_demo/postprocessor.hpp similarity index 100% rename from demos/multichannel_demo/hpe/postprocessor.hpp rename to demos/multi_channel/human_pose_estimation_demo/postprocessor.hpp diff --git a/demos/multichannel_demo/hpe/render_human_pose.cpp b/demos/multi_channel/human_pose_estimation_demo/render_human_pose.cpp similarity index 95% rename from demos/multichannel_demo/hpe/render_human_pose.cpp rename to demos/multi_channel/human_pose_estimation_demo/render_human_pose.cpp index 4ddf4744da0..4ecf53ca5d3 100644 --- a/demos/multichannel_demo/hpe/render_human_pose.cpp +++ b/demos/multi_channel/human_pose_estimation_demo/render_human_pose.cpp @@ -19,12 +19,13 @@ #include +#include "postprocess.hpp" #include "render_human_pose.hpp" void renderHumanPose(const std::vector& poses, cv::Mat& image) { CV_Assert(image.type() == CV_8UC3); - const std::vector colors = { + static const cv::Scalar colors[keypointsNumber] = { cv::Scalar(255, 0, 0), cv::Scalar(255, 85, 0), cv::Scalar(255, 170, 0), cv::Scalar(255, 255, 0), cv::Scalar(170, 255, 0), cv::Scalar(85, 255, 0), cv::Scalar(0, 255, 0), cv::Scalar(0, 255, 85), cv::Scalar(0, 255, 170), @@ -32,7 +33,7 @@ void renderHumanPose(const std::vector& poses, cv::Mat& image) { cv::Scalar(0, 0, 255), cv::Scalar(85, 0, 255), cv::Scalar(170, 0, 255), cv::Scalar(255, 0, 255), cv::Scalar(255, 0, 170), cv::Scalar(255, 0, 85) }; - const std::vector > limbKeypointsIds = { + static const std::pair limbKeypointsIds[] = { {1, 2}, {1, 5}, {2, 3}, {3, 4}, {5, 6}, {6, 7}, {1, 8}, {8, 9}, {9, 10}, diff --git a/demos/multichannel_demo/hpe/render_human_pose.hpp b/demos/multi_channel/human_pose_estimation_demo/render_human_pose.hpp similarity index 100% rename from demos/multichannel_demo/hpe/render_human_pose.hpp rename to demos/multi_channel/human_pose_estimation_demo/render_human_pose.hpp diff --git a/demos/multi_channel/object_detection_demo_yolov3/CMakeLists.txt b/demos/multi_channel/object_detection_demo_yolov3/CMakeLists.txt new file mode 100644 index 00000000000..1490e4242c7 --- /dev/null +++ b/demos/multi_channel/object_detection_demo_yolov3/CMakeLists.txt @@ -0,0 +1,79 @@ +# Copyright (C) 2018-2019 Intel Corporation + +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at + +# http://www.apache.org/licenses/LICENSE-2.0 + +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set(TARGET_NAME "multi_channel_object_detection_demo_yolov3") + +if( BUILD_DEMO_NAME AND NOT ${BUILD_DEMO_NAME} STREQUAL ${TARGET_NAME} ) + message(STATUS "DEMO ${TARGET_NAME} SKIPPED") + return() +endif() + +# Find OpenCV components if exist +find_package(OpenCV COMPONENTS highgui QUIET) +if(NOT(OpenCV_FOUND)) + message(WARNING "OPENCV is disabled or not found, " ${TARGET_NAME} " skipped") + return() +endif() + +file (GLOB MAIN_SRC + ${CMAKE_CURRENT_SOURCE_DIR}/*.cpp + ) + +file (GLOB MAIN_HEADERS + ${CMAKE_CURRENT_SOURCE_DIR}/*.hpp + ) + +# Create named folders for the sources within the .vcproj +# Empty name lists them directly under the .vcproj +source_group("src" FILES ${MAIN_SRC}) +source_group("include" FILES ${MAIN_HEADERS}) + +# Create library file from sources. +add_executable(${TARGET_NAME} ${MAIN_SRC} ${MAIN_HEADERS}) + +set_target_properties(${TARGET_NAME} PROPERTIES + POSITION_INDEPENDENT_CODE ON + COMPILE_PDB_NAME ${TARGET_NAME}) + +if(MULTICHANNEL_DEMO_USE_TBB) + find_package(TBB REQUIRED tbb) + target_link_libraries(${TARGET_NAME} ${TBB_IMPORTED_TARGETS}) + target_compile_definitions(${TARGET_NAME} PRIVATE + USE_TBB=1 + __TBB_ALLOW_MUTABLE_FUNCTORS=1) + + if(FALSE) # disable task isolation for now due to bugs in tbb + target_compile_definitions(${TARGET_NAME} PRIVATE + TBB_PREVIEW_TASK_ISOLATION=1 + TBB_TASK_ISOLATION=1) + endif() +endif() + +target_link_libraries(${TARGET_NAME} ${InferenceEngine_LIBRARIES} gflags ${OpenCV_LIBRARIES} common) + +if(UNIX) + target_link_libraries( ${TARGET_NAME} pthread) +endif() + +if(COMMAND add_cpplint_target) + add_cpplint_target(${TARGET_NAME}_cpplint FOR_TARGETS ${TARGET_NAME}) +endif() + +if(NOT TARGET ie_samples) + add_custom_target(ie_samples ALL) +endif() + +add_dependencies(ie_samples ${TARGET_NAME}) + +target_link_libraries(${TARGET_NAME} monitors) diff --git a/demos/multi_channel/object_detection_demo_yolov3/README.md b/demos/multi_channel/object_detection_demo_yolov3/README.md new file mode 100644 index 00000000000..636d5049e70 --- /dev/null +++ b/demos/multi_channel/object_detection_demo_yolov3/README.md @@ -0,0 +1,114 @@ +# Multi-Channel Object Detection Yolov3 C++ Demo + +This demo provides an inference pipeline for multi-channel yolo v3. The demo uses Yolo v3 Object Detection network. You can follow [this](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_YOLO_From_Tensorflow.html) page convert the YOLO V3 and tiny YOLO V3 into IR model and execute this demo with converted IR model. + +> **NOTES**: +> If you don't use [this](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_YOLO_From_Tensorflow.html) page to convert the model, it may not work. + +Other demo objectives are: + +* Up to 16 cameras as inputs, via OpenCV* +* Visualization of detected objects from all channels on a single screen + + +## How It Works + +On the start-up, the application reads command line parameters and loads the specified networks. The Yolo v3 Object Detection network is required. + +> **NOTES**: +> * By default, Open Model Zoo demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the demo application or reconvert your model using the Model Optimizer tool with `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model Using General Conversion Parameters](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html). + +## Running + +Running the application with the `-h` option yields the following usage message: +``` +cd /intel64/Release +./multi_channel_object_detection_demo_yolov3 -h + +multi_channel_object_detection_demo_yolov3 [OPTION] +Options: + + -h Print a usage message. + -m "" Required. Path to an .xml file with a trained yolo v3 or tiny yolo v3 model. + -l "" Required for MKLDNN (CPU)-targeted custom layers. Absolute path to a shared library with the kernels impl. + Or + -c "" Required for clDNN (GPU)-targeted custom kernels. Absolute path to the xml file with the kernels desc. + -d "" Optional. Specify the target device for Face Detection (CPU, GPU, FPGA, HDDL or MYRIAD). The demo will look for a suitable plugin for a specified device. + -nc Optional. Maximum number of processed camera inputs (web cams) + -bs Optional. Batch size for processing (the number of frames processed per infer request) + -nireq Optional. Number of infer requests + -n_iqs Optional. Frame queue size for input channels + -fps_sp Optional. FPS measurement sampling period. Duration between timepoints, msec + -n_sp Optional. Number of sampling periods + -pc Optional. Enables per-layer performance report. + -t Optional. Probability threshold for detections. + -no_show Optional. No show processed video. + -show_stats Optional. Enable statistics report + -duplicate_num Optional. Enable and specify number of channel additionally copied from real sources + -real_input_fps Optional. Disable input frames caching, for maximum throughput pipeline + -i Optional. Specify full path to input video files + -u Optional. List of monitors to show initially. +``` + +To run the demo, you can use public pre-train model and follow [this](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_YOLO_From_Tensorflow.html) page for instruction of how to convert it to IR model. + +> **NOTE**: Before running the demo with a trained model, make sure the model is converted to the Inference Engine format (\*.xml + \*.bin) using the [Model Optimizer tool](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html). + +For example, to run the demo on FPGA with fallback on CPU, with one single camera, use the following command: +```sh +./multi_channel_object_detection_demo_yolov3 -m $PATH_OF_YOLO_V3_MODEL -d HETERO:FPGA,CPU -nc 1 +``` + +To run the demo using two recorded video files, use the following command: +```sh +./multi_channel_object_detection_demo_yolov3 -m $PATH_OF_YOLO_V3_MODEL -d HDDL -i /path/to/file1 /path/to/file2 +``` +Video files will be processed repeatedly. + +To achieve 100% utilization of one Myriad X, the thumb rule is to run 4 infer requests on each Myriad X. Option `-nireq 32` can be added to above command to use 100% of HDDL-R card. The 32 here is 8 (Myriad X on HDDL-R card) x 4 (infer requests), such as following command: + +```sh +./multi_channel_object_detection_demo_yolov3 -m $PATH_OF_YOLO_V3_MODEL -d HDDL +-i /path/to/file1 /path/to/file2 /path/to/file3 /path/to/file4 -nireq 32 +``` + +You can also run the demo on web cameras and video files simultaneously by specifying both parameters: `-nc -i