VQ Probe v2.0 User Guide
- Graphical User Interface Overview
- Video Quality Metrics
- Command Line Interface
- Known Limitations
- Third Party Libraries
VQ Probe is a software tool for Video Elementary Stream visual comparison.
The tool is shipped as a Graphical User Interface application for Windows, macOS, and Linux.
Supported Video Codecs
- H.266/VVC VTM-10.0 (4:2:0 only)
Supported Uncompressed Formats
Supported Chroma Subsampling
Supported Bit Depth
- 8 bit
- 10 bit
Supported Video Metrics
- PSNR: calculated based on the regular formula (Peak signal-to-noise ratio); average noise of every pixel of the whole frame is calculated
- SSIM: we compute SSIM using the empirical downsampling process. For each image let F = max(1, round(N/256)), where N is the height of an image in pixels. The image is averaged within an F by F window and downsampled by a factor of F. Then the SSIM index is calculated by the following formula:
- C1 and C2 are constants equal to (k1L)2 and (k2L)2, respectively, where by default k1 = 0.01, k2 = 0.03, and L is the dynamic range of the images;
- μ1 and μ2 are the compared windows;
- σ1 is the difference between the first image squared and (μ1)2;
- σ2 is the difference between the second image squared and (μ2)2;
- σ12 is the difference between the multiplication of the images and μ1μ2.
- VMAF: the algorithm and the model v0.6.1 (https://github.com/Netflix/vmaf) are used for VMAF calculation. Please follow VMAF: The Journey Continues for more information.
When you launch VQ Probe for the first time, you will be asked to enter credentials provided to you with your copy of the application. Enter the given IP and port to register your copy of VQ Probe. This is only needed once.
If you want to upgrade your copy of VQ Probe, go to Help > Activate, enter your new credentials, press the Activate button, and restart the application.
Graphical User Interface Overview
The main window of VQ Probe has the following elements:
- Project View
- Playback View
- Quality Metrics View
The window used to set up and manage your workspace: add and remove video files, select which ones will be displayed in Playback View, create and remove RD Curves, set the RD Curve anchor used for BD Rate calculation, inspect the left and the right files in Buffer View.
The view can be shown or hidden by using Window > Project View from the main menu, or by using hotkeys (Ctrl + Shift + P on Windows and Linux, or Cmd + Shift + P on macOS).
- Left file area displays a part of the current frame of the left file
- Right file area displays a part of the current frame of the right file
- Vertical split line is placed between two images; it allows the user to change the width of the left and the right images
- Buttons to choose which component(s) are shown:
- YUV: show YUV
- Y: show only Y plane
- U: show only U plane
- V: show only V plane
- Buttons to choose the format of numbers in Playback View
- dec: show numbers as decimal
- hex: show numbers as hexadecimal
- View mode:
- L|R: show the left and the right files divided by the splitter line
- Ref: show the reference file
- L|R / Ref Diff: show the difference between the left and the reference files on the left side of the screen and the difference between the right and the reference files on the right side of the screen
- B/W Diff: show the difference just like L|R / Ref Diff, but use only two colors (white if there is a difference and black if there is no difference)
- Navigation buttons:
- First frame: go to the first frame
- Previous frame: go to the previous frame
- Play/pause: start playback or pause, respectively
- Next frame: go to the next frame
- Last frame: go to the last frame
All buttons are duplicated by the appropriate items in the Playback menu. Hotkeys are also available for the buttons listed above. Please follow the main menu for exact hotkeys depending on the Operating System you use.
If you hover your mouse over the frame, you will see the mouse coordinates (in pixels) in the upper left corner.
NOTE: The left or right area might not display any frame after navigation. This means that there is no corresponding frame in the appropriate file while the other file has a frame with the same frame number. This happens if opened files have different numbers of frames.
Quality Metrics View
The view consists of the following elements:
- Four tabs:
- Metrics: contains regular metric values for each file
- L/R Quality Diff: shows the difference between metrics of the left file and the right file
- RD Curves: contains RD Curves (if exist) as well as BD Rate information
- L/R Object Count: shows the number of objects found for the left file and the right file
- Metric combo-box to choose a metric to display on the main area. It could be one of the following:
- PSNR YUV, PSNR Y, PSNR U, PSNR V
- SSIM YUV, SSIM Y, SSIM U, SSIM V
- VMAF YUV, VMAF Y, VMAF U, VMAF V
- Run/Stop buttons are to start or stop, respectively, metrics calculation
- Three ROI buttons ((see the Region of Interest section)
- The Log button to show the Metrics Log
The Run button is disabled if a reference file is not opened. Once the user clicks the Run button, it is replaced by the Stop button. And vice versa: once the user clicks Stop, or when the metrics calculation process is done, Run appears again, and replaces the Stop button.
Quality Metrics View can be hidden or shown by checking or unchecking Window > Metrics View, or by using hotkeys (Ctrl + Shift + M on Windows and Linux, or Cmd + Shift + M on macOS).
Once VQ Probe is launched, it has an empty project by default. You may open a project saved before, or, you can set up the project for your needs from scratch. To do so, start by adding source files.
Add Source Files
By source files we assume video files that are used in analysis and metrics calculation, as well as any other operations within VQ Probe. These could be uncompressed files as well as encoded files.
Source files can be added to the current project in several ways:
- Via Project View
- By pressing the "+" button on the Source Files panel of Project View. The standard system "Open Files" dialog will be opened. Select one or more files and press the Open button. Selected files will be analyzed then. If the files are valid and supported by VQ Probe, they will be added to the list of files. Otherwise, a message box with an error description will appear.
- By dropping files on Playback View. Files will be added to Project View
Now, when files are successfully opened, their information is displayed on the screen: codec, resolution, bitrate, framerate (the default value is 25), and frame count.
NOTE: Files must have the same resolution as well as the same chroma subsampling format.
Possible reasons why files cannot be added are:
- the files are not supported (please check the appropriate section for the list of supported formats)
- the files have different resolution and/or chroma subsampling format
- an internal application error
Remove Source Files
Select one or more files from the list on the Source Files tab and press the right mouse button. In the opened menu choose Remove.
Select Files Displaying
Use checkboxes from the Left, Right, or Ref columns to set the appropriate file to be displayed as the left, the right, or the reference file, respectively, on Playback View:
Create RD Curves
Press the right mouse button at any empty space or on any RD Curve item on the RD Curves tab. Choose Add RD Curve from the opened menu. Enter a name for a new RD Curve and press OK.
You cannot choose the name of any existing RD Curve. You cannot either choose the name of any already existed RD Curve. In both cases, you will be prompted to choose another name.
Remove RD Curves
Press the right mouse button on one or more RD Curves on the RD Curves tab, then choose Remove from the opened menu.
Add Source Files to RD Curve
Select one or more source files on the Source Files tab, then select the Add to RD Curve menu item. You will see the list of RD Curves where it's allowed to add the selected source files:
The selected source files from the example above cannot be added to AVC nor HEVC RD Curves because they are already added.
NOTE: A reference file cannot be added to an RD Curve. YUV files cannot be added as well.
Once you add source files to an RD Curve, they will appear in the list of RD Curves:
Remove Source Files from RD Curve
Choose one or more source files on the RD Curve tab, press the right mouse button, and choose Remove:
All the selected source files will be removed, even if they are referring to different RD Curves.
Set RD Curve as Anchor
An anchor RD Curve is used to calculate BD Rate. BD Rate will be calculated for all other RD Curves and relatively to the one marked as an anchor RD Curve.
To mark an RD Curve as the anchor, select that RD Curve and press the right mouse button:
Once marked, its name will be changed:
Buffer View presents frame sizes of the left and the right files. This is a double bar chart (the left file's bars are on top) plotted with frame sizes. Each bar is colored according to the frame type.
Move the mouse over a bar to get a tooltip with information about a frame represented by this bar. The current frame of Playback View is shown with a vertical line and a frame number on it.
Use the top graph for navigation. Double clicking on the canvas area resets the graphs back to the default settings. You can zoom in and out with the mouse wheel on both axis when the cursor is inside the canvas.
By double clicking on the Bitrate, kbps [25fps] label, the fps value can be adjusted in the separate dialog box.
You can move the legend by pressing the left mouse button and moving the mouse. You can also click one of the legend items to show or hide the corresponding bar/curve. Click the right mouse button to restore the default position of the legend.
NOTE: Frames appear in decoding order. Every tooltip of a bar starts with two numbers: x/y, where x is the frame number in decoding order and y is the frame number in display order.
Opening YUV Files
Y4M files are opened automatically. In case of a regular YUV file, the user will be asked to specify video parameters, such as resolution, chroma subsampling, whether it's packed or planar, and bit depth.
Configure parameters of the file. Once a parameter is changed, the preview image updates. So, you can tune parameters until the image looks right to you.
NOTE: Once the user opened a specific file via the dialog, consequent openings of the same file will use the previous parameters as the default parameters.
For YUV/Y4M files you have the option to crop the image. This can be done in Open YUV File Dialog:
There are 4 parameters to set up cropping:
- Left: position in pixels where to start cropping horizontally
- Top: position in pixels where to start cropping vertically
- Width: width of the cropped area in pixels
- Height: height of the cropped area in pixels
Once any of these parameters are changed, the cropped area visualization will be updated on the left side of the dialog.
NOTE: Cropping parameters will be automatically changed to the nearest multiple of 4 or 8 depending on the Plane Order. You may find the details under the information sign of the dialog.
You can trim source files by double-clicking on the Start or the End column for a specific source file. A spin box will appear:
Type in or select a required frame number and press Enter, or click the left mouse button outside the spin box. The value will be changed.
All first frame/last frame numbers that have been changed are marked in orange. For example, the first two source files below have the values that have been changed:
You can apply trimming to all source files. To do that, press the right mouse button on the Start or the End column for a specific source file and then choose either Apply Start Frame Number to Others or Apply End Frame Number to Others:
Once the option is chosen, all the files will be trimmed:
NOTE: Trimming source files will affect the frame numbering used in Playback View and Metrics View. For example, if a source file has been trimmed and now starts from Frame 10, then the frame marked as Frame 0 in Metrics View will correspond to Frame 10 from the original source file.
Playback can be started by clicking the appropriate Play button, by choosing the menu item Playback > Play/Pause or by using the Space hotkey. To stop playback, press the Pause button, choose the menu item Playback > Play/Pause or use the Space hotkey.
Playback can be looped by checking the Playback > Loop menu item or by using the Ctrl + L hotkey. By default, playback is not looped.
Framerate used for playback can be set to one of the values placed under the Playback > Framerate menu. The default value is 25 fps.
NOTE: Depending on hardware resources, video framerate of playback can be less than targeted in the menu.
Overlap View / Independent View
By default, VQ Probe is in the Overlap View mode. You can switch to the Independent View mode. In this mode, the left and the right files are shown in full, side by side, and the splitter line is not movable.
To change the current mode, click the View > Split Line Mode submenu and choose either Overlap View or Independent View.
YUV values of the opened files become available starting at a zoom ratio of 32x. These values will look as follows:
If the current visible component is chosen for displaying one plane (Y, U, or V), the values for that specific component will be shown. For example:
Use the dec/hex switch to show values in decimal or hexadecimal format, respectively.
Heat Map: Difference between Left(Right) and Reference
This mode allows you to visualize in the form of a heat map the difference between the left file's specific frame and the appropriate frame of the reference file as well as the difference between the right file's specific frame and the appropriate frame of the reference file.
Please choose L|R / Ref Diff (or View > Show Left/Right and Reference Difference from the main menu, or press F3 on your keyboard) to show the heat map. It looks as follows:
The color represents the difference between a specific pixel of the left (right) frame and a specific pixel of the reference frame, from 0 to the maximum difference:
For example, black color means there is no difference.
Once zoomed in to a zoom ratio of 32x and above, the values of the difference will be shown:
You may also choose a specific plane (Y, U, or V) to show the difference of a specific plane.
Heat Map: B/W Difference
This mode is similar to the previous one: the only difference is that in this mode VQ Probe uses only two colors: white (if there is a difference) and black (if there is no difference).
Please choose B/W Diff (or View > Show B/W Difference from the main menu, or press F4 on your keyboard) to show the black and white heat map. It looks as follows:
Again, once zoomed in to a zoom ratio of 32x and above, the values of the difference will be shown:
Source files can be zoomed in as well as zoomed out. Following are the options:
- choose the menu items View > Zoom In and View > Zoom Out to zoom in and zoom out, respectively
- use the keyboard shortcuts
Specific key combinations depend on the Operating System the application is running on. You may find the exact shortcut combination in the View menu.
- press Ctrl and the left mouse button to start selection of the area to zoom in. The source file will be zoomed in once the mouse button is released
There are some additional convenient functions:
- change the size to the actual video resolution by using View > Actual Size
- fit the source file to the window's size by choosing View > Fit to Window in the menu
NOTE: In case when the source file is shifted from the center of Playback View, using the keyboard shortcuts may shift it even more.
Video Quality Metrics
VQ Probe allows the user to calculate video quality metrics. The list of quality metrics is provided in section Supported Video Metrics.
The functionality of video quality metrics calculation is available on Metrics View.
Once at least one target file is opened, and the reference file is opened, VQ Probe is ready to calculate video quality metrics. The user can do that in one of the following ways:
- by clicking Metrics > Run in the main menu
- by clicking Run in Metrics View
- by using hotkeys: Cmd + M, Cmd + R (macOS) or Ctrl + M, Ctrl + R (Windows, Linux)
Next, metrics calculation starts for every available metric: PSNR, SSIM, and VMAF. Values appear immediately:
The Run button is disabled if there is no reference file opened.
The Run button is replaced by the Stop button during the metrics calculation process. When the metrics calculation process is in progress, the Pause button appears. By pressing this button you can pause the calculation. When paused, the Resume button appears instead of the Pause button. If you click the Resume button, the calculation will resume. You can also use the corresponding actions in the Metrics menu or the Ctrl + Space hotkey.
The currently visible metric can be changed via the Metric combo-box.
For every graph a legend is available. You can move it by pressing the left mouse button and moving the mouse. Click the right mouse button to restore the default position of the legend.
You can also click one of the legend items to show or hide the corresponding curve.
Region of Interest
The user can set a region of interest (or ROI). A ROI is a rectangle set for all streams in a project, which is used to calculate metrics (that is, metrics are calculated only for this region).
Use the Set ROI button to specify a file in the COCO JSON format containing the desirable ROI (we are using the annotations/bbox field to set the ROI; read this specification for more information).
Click the Current ROI button to view the current ROI file. The Unset ROI button is used to reset the ROI.
Please note that a region of interest must be a valid rectangle that completely fits into the frame rectangle. Its width and height must be at least 15 pixels long.
Check the View > Show ROI Bounding Boxes menu item to make the current ROI appear on Playback View.
Please note that a ROI is set for a project; when you close the current project or remove all the source files, the current ROI will be reset.
VQ Probe has the ability to present the results of metrics calculation not only in a form of graphs, but also using the Metrics Log window. Click the Log button to open it.
The Metrics Log consists of tables for every non-reference file of the project. Every table contains metrics values for every frame of a stream. Click the Export button to export the current table.
Once you check the Autosave Results check box (found here: Metrics > Options > General; see the Options section), the results of metrics calculation will be saved in the project file after the calculation process is finished.
If you also check the Use Cached Results check box, the next time you press the Run button, graphs will be built on the fly using the cached data.
NOTE: If your project has some unsaved changes and the Autosave Results check box is checked, these changes will be saved anyway the next time the results are saved.
The user can see exact numbers for a specific frame by moving the cursor over a graph. A tooltip will show the frame number and metric values for that frame.
You can navigate by clicking on a specific frame in Metrics View. After that, the actual image(s) in Playback View will be changed according to the selected frame.
Similarly, once the current frame is changed in Playback View, the current frame selection in Metrics View will be changed.
The L/R Quality Diff tab allows the user to get the difference between the metrics of the right file and the metrics of the left file. In other words, it shows how much the right file is better than the left file:
The values on the graph will be negative if the right file is of a lower quality than the left file. For example:
RD Curves tab allows you to plot RD Curves as well as calculate BD Rates. Please make sure you set up the RD Curves list (please refer to the Project section).
The tab contains the following information:
- BD Rates: the list of BD Rates. The first RD Curve is assumed as the anchor. BD Rates for other RD Curves are calculated in relation to the anchor RD Curve
- RD Curve graph: a graph that consists of points, each of them is an average appropriate metric value at a specific file's average bitrate
Changing the Metric combo-box will change the current RD Curves graph to a graph for the selected metric.
Please note that BD Rate calculation is undefined in several cases: - RD Curves contain different number of files - RD Curves contain only one file
NOTE: For the case of two-file RD Curves, the accuracy of BD Rate calculation will be rather low.
Convex Hull for RD Curves
In this mode, a convex hull will be built for the given set of RD Curves. To enable this mode, check the Enable Convex Hull check box in Project View > RD Curves.
When the mode is enabled, you can add streams of different resolutions. It can be useful to set up a couple of RD Curves (each curve consisting of streams with the same resolution), then run the calculation. The resulting convex hull can be used to compare different resolutions of the reference file.
Another feature of VQ Probe is the ability to detect objects using machine learning. The results are presented on the L/R Object Count tab.
Click the Run button to start the detection. You can also pause the detection (and then resume it). If you click the Stop button, the detection will be stopped.
Use the Log button to see the output of the object detection model (i.e. found objects, scores, etc.). The objects presented on the graph are highlighted in green.
Check the View > Show Object Bounding Boxes menu item to make the objects presented on the graph appear on Playback View.
Scene Change Detection
VQ Probe is capable of finding scene changes in a stream that is set either as the left, the right, or the reference file (see the Options section). To enable scene change detection, check the Playback > Show Scene Changes menu item or use the Ctrl + E hotkey.
Please note that you may need to reopen the current project to trigger the detection.
Go to Metrics > Options or use the Ctrl + Shift + O hotkey to open the Metrics Options dialog.
There are 3 tabs:
- Autosave Results: autosave the results of video quality metrics calculation
- Use Cached Results: use the results stored in the project file to build graphs on the fly
- Results Folder: set the results folder
- Clear Results Folder: delete all files from the results folder
- Object Detection:
- Score Threshold: set the score threshold (i.e. objects with a score below this threshold will not be presented on the L/R Object Count graph). The default value is 0,75.
- Choose the active classes: only objects with a class that is set as active will be presented
Please note that the Object Detection tab is not accessible during the detection. Click the Stop button or wait for the detection to finish to access the tab.
- Scene Change Detection:
- Show Scene Changes For: select the file (L, R, or Ref) for which scene changes will be detected
- Threshold: set the threshold (the default value is 0,05)
- Scene Change Colors: choose the colors that indicate scenes on the slider
To restore the default options, use Metrics > Restore Default Options.
Command Line Interface
Once you activate your copy of VQ Probe, you can use the Command Line Interface (CLI) of the app. You can find it near the main executable of VQ Probe (in VQProbe.app/Contents/MacOS on macOS). The CLI executable is called VQProbeConsole.
The CLI is used to calculate metrics and store them in our export format. These are the options of the CLI:
- -h: displays help information
- -v: displays version information
- -p: calculate metrics for the
- -d: save the calculated metrics to
- --dont-update: don't update the project file (i.e. dont set the new results as the project's cache)
- While using 4K+ resolution, the "+" button in Project View is too small.
- If you want to set streams with different resolutions as the left, the right, and the reference files, you must set a stream with the highest resolution as the left.
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH VICUE SOFT PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN VICUE SOFT'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, VICUE SOFT ASSUMES NO LIABILITY WHATSOEVER AND VICUE SOFT DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF VICUE SOFT PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY VICUE SOFT, THE VICUE SOFT PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE VICUE SOFT PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.
ViCue Soft may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." ViCue Soft reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Contact your local ViCue Soft sales office or your distributor to obtain the latest specifications and before placing your product order.
MPEG is an international standard for video compression/decompression promoted by ISO. Implementations of MPEG CODECs, or MPEG enabled platforms may require licenses from various entities, including ViCue Soft.
*Other names and brands may be claimed as the property of others. Copyright © 2020-2021, ViCue Soft. All Rights reserved.
Third Party Libraries
VQ Probe uses certain 3rd party libraries, as listed below. You may find the full text of each license in the following folder: Windows/Linux – the licenses folder under your application's folder; macOS – VQProbe.app/Contents/SharedSupport/licenses.
Qt is licensed under the GNU Lesser General Public License (LGPL) version 3.
Project homepage https://www.qt.io/, upstream version: 5.13.1.
Copyright (C) 2019 The Qt Company Ltd and other contributors
You may find the full text of the license in the Qt_license file of your binary package.
Qwt is licensed under the Qwt License, Version 1.0.
Project homepage https://qwt.sourceforge.io/, upstream version: 6.1.4.
Copyright (c) 2019 Uwe Rathmann
You may find the full text of the license in the qwt_license file of your binary package.
FFmpeg is licensed under the GNU Lesser General Public License (LGPL) version 2.1.
Project homepage https://ffmpeg.org, upstream version: 4.2.2.
Copyright (c) 2000-2019 the FFmpeg developers
You may find the full text of the license in the ffmpeg_license file of your binary package.
Libvmaf is licensed under the BSD+Patent License.
Project homepage https://github.com/Netflix/vmaf, upstream version: 1.3.16.
Copyright (c) 2020 Netflix, Inc.
You may find the full text of the license in the libvmaf_license file of your binary package.
VVCSoftware_VTM is licensed under the BSD License.
Project homepage https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/tree/VTM-10.0, upstream version: 10.0.
Copyright (c) 2010-2020, ITU/ISO/IEC
You may find the full text of the license in the libvvc_license file of your binary package.
dav1d is licensed under the BSD 2-clause "Simplified" License.
Project homepage https://code.videolan.org/videolan/dav1d, upstream version: 0.7.1.
Copyright © 2018-2019, VideoLAN and dav1d authors
You may find the full text of the license in the dav1d_license file of your binary package.