Skip to content

Commit d72e4cb

Browse files
VladimirUmekVladimirUmek
authored
vStream: enhance Audio and Video capabilities (#36)
- Add microphone handling - Add streaming from and to audio file (wav only) - Add audio/video driver configuration - Simplify FILENAME register handling - Add DEVICE register to control streaming device by index
1 parent 35e0ed5 commit d72e4cb

24 files changed

+2321
-757
lines changed

‎ARM.AVH_FVP.pdsc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@
5050
<file category="doc" name="Documentation/simulation/html/group__arm__vsi.html"/>
5151
<file category="include" name="interface/include/"/>
5252
<file category="sourceC" name="interface/vstream/source/vstream_audio_in.c"/>
53+
<file category="header" name="interface/vstream/source/vstream_audio_in.h"/>
54+
<file category="header" name="interface/vstream/config/vstream_audio_in_config.h" attr="config" version="1.0.0"/>
5355
</files>
5456
</component>
5557

@@ -62,6 +64,8 @@
6264
<file category="doc" name="Documentation/simulation/html/group__arm__vsi.html"/>
6365
<file category="include" name="interface/include/"/>
6466
<file category="sourceC" name="interface/vstream/source/vstream_audio_out.c"/>
67+
<file category="header" name="interface/vstream/source/vstream_audio_out.h"/>
68+
<file category="header" name="interface/vstream/config/vstream_audio_out_config.h" attr="config" version="1.0.0"/>
6569
</files>
6670
</component>
6771

‎interface/vstream/README.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# vStream via VSI
2+
3+
This directory contains the implementation of vStream interface for audio and video streaming
4+
using Virtual Streaming Interface (VSI). The vStream interface is defined by CMSIS-Driver and
5+
offers standardized interface to interact with various streaming devices to allow seamless
6+
development on one platform and deployment on another.
7+
8+
Implementation consists of two components
9+
10+
- vStream driver implementation in C (targeting embedded application)
11+
- implementation of VSI Python interface in Python (targeting host PC)
12+
13+
## vStream Drivers
14+
15+
Directory `source` contains CMSIS-Driver vStream implementations for audio input/output
16+
and video input/output streaming.
17+
18+
**Audio Drivers**:
19+
20+
- `vstream_audio_in.c/h` - vStream_AudioIn implementation
21+
- `vstream_audio_out.c/h` - vStream_AudioOut implementation
22+
23+
**Video Drivers**:
24+
25+
- `vstream_video_in.c/h` - vStream_VideoIn implementation
26+
- `vstream_video_out.c/h` - vStream_VideoOut implementation
27+
28+
### Configuration for vStream Drivers
29+
30+
Directory `config` contains configuration headers for vStream drivers.
31+
32+
Configuration enables user to configure basic audio and video stream parameters
33+
to adapt vStream driver to embedded application expectation.
34+
35+
**Configuration Files**:
36+
37+
- `vstream_audio_in_config.h` - configures vStream_AudioIn
38+
- `vstream_audio_out_config.h` - configures vStream_AudioOut
39+
- `vstream_video_in_config.h` - configures vStream_VideoIn
40+
- `vstream_video_out_config.h` - configures vStream_VideoOut
41+
42+
**Audio Stream Configuration**:
43+
44+
Configuration options for configuring vStream audio drivers are:
45+
46+
| Parameter | Description | Available Options | Configuration Macro |
47+
|---------------------------|------------------------------------------------------|---------------------------------|------------------------------|
48+
| Number of channels | Defines the number of audio channels in stream | Mono (1), Stereo (2) | `AUDIO_[IN/OUT]_CHANNELS` |
49+
| Number of bits per sample | Defines number of bits of information in each sample | 8-bit, 16-bit, 24-bit, 32-bit | `AUDIO_[IN/OUT]_SAMPLE_BITS` |
50+
| Sample rate | Defines the number of samples per second | 8 kHz, 16 kHz, 44.1 kHz, 48 kHz | `AUDIO_[IN/OUT]_SAMPLE_RATE` |
51+
52+
**Video Stream Configuration**:
53+
54+
Configuration options for configuring vStream Video drivers are:
55+
56+
| Parameter | Description | Available Options | Configuration Macro |
57+
|--------------|----------------------------------------------------------|---------------------------------|-------------------------------|
58+
| Frame width | Defines the video stream frame width in pixels | User-defined (e.g., 320, 640) | `VIDEO_[IN/OUT]_FRAME_WIDTH` |
59+
| Frame height | Defines the video stream frame height in pixels | User-defined (e.g., 240, 480) | `VIDEO_[IN/OUT]_FRAME_HEIGHT` |
60+
| Frame rate | Defines the video stream frame rate in frames per second | User-defined (e.g., 15, 30, 60) | `VIDEO_[IN/OUT]_FRAME_RATE` |
61+
| Color format | Defines the video frame color space | Grayscale, RGB888, BGR565 | `VIDEO_[IN/OUT]_FRAME_COLOR` |
62+
63+
**Common Configuration**:
64+
65+
Besides stream configuration there are configuration options related to stream source selection. The below table shows possible stream sources:
66+
67+
| Stream Type | Stream Source | Description | Configuration |
68+
|---------------|-----------------------|-----------------------------------------------|------------------------------------------|
69+
| **Audio In** | System Default Input | Uses the system's default audio input device | `AUDIO_IN_DEVICE = -1` |
70+
| | Specific Audio Device | Uses a specific audio input device by index | `AUDIO_IN_DEVICE = <device_index>` |
71+
| | Audio File | Reads audio data from a file | `AUDIO_IN_FILENAME = "path/to/file.wav"` |
72+
| **Audio Out** | System Default Output | Uses the system's default audio output device | `AUDIO_OUT_DEVICE = -1` |
73+
| | Specific Audio Device | Uses a specific audio output device by index | `AUDIO_OUT_DEVICE = <device_index>` |
74+
| | Audio File | Writes audio data to a file | `AUDIO_OUT_FILENAME = "path/to/file.wav"`|
75+
| **Video In** | System Default Input | Uses the system's default video input device | `VIDEO_IN_DEVICE = -1` |
76+
| | Specific Input Device | Uses a specific camera device by index | `VIDEO_IN_DEVICE = <device_index>` |
77+
| | Video File | Reads video frames from a file | `VIDEO_IN_FILENAME = "path/to/file.mp4"` |
78+
| **Video Out** | Display Window | Displays video frames in a window | No device configuration needed |
79+
| | Video File | Writes video frames to a file | `VIDEO_OUT_FILENAME = "path/to/file.mp4"`|
80+
81+
> [!NOTE]
82+
>
83+
> OpenCV is not able to automatically detect system default device. Instead it enumerates devices, for example:
84+
>
85+
> - 0 being the internal camera
86+
> - 1 is the first external camera
87+
88+
## Virtual Streaming Interface for Python
89+
90+
Directory `python` contains Python scripts that implement [VSI for Python](https://arm-software.github.io/AVH/main/simulation/html/group__arm__vsi__py.html)
91+
and execute on host PC. Audio and video servers are implemented independently to support access to audio and video streams.
92+
93+
**VSI scripts for Python**:
94+
95+
The `arm_vsi*.py` files provide VSI implementations for different channels.
96+
97+
- `arm_vsi0.py` and `arm_vsi1.py` implement Audio streaming channels
98+
- `arm_vsi4/vsi5/vsi6/vsi7.py` - implement Video streaming channels
99+
100+
**Audio Server**:
101+
102+
- `vsi_audio_server.py` and
103+
- `vsi_audio.py`
104+
105+
implement Python server and client for audio streaming using PyAudio.
106+
107+
**Video Server**:
108+
109+
- `vsi_video_server.py` and
110+
- `vsi_video.py`
111+
112+
implement Python server and client for video streaming using OpenCV.
113+
114+
### Audio Server
115+
116+
The Audio Server implements a TCP-based server for audio streaming operations using PyAudio. It provides comprehensive audio I/O capabilities for both real-time audio devices and file-based streaming.
117+
118+
**Key Features**:
119+
120+
- Supports both audio input (microphone/file recording) and audio output (speaker/file playback)
121+
- Supports automatic detection and configuration of system audio devices
122+
- Supports WAV file reading and writing
123+
- Allows multiple client connections
124+
125+
### Video Server
126+
127+
The Video Server implements a TCP-based server for video streaming operations using OpenCV. It provides video I/O capabilities for accessing cameras and video files.
128+
129+
**Key Features**:
130+
131+
- Supports both video input (camera/file capture) and video output (display/file recording)
132+
- Supports access to system cameras and USB video devices
133+
- Supports multiple video formats (AVI, MP4, WMV) and image formats (BMP, PNG, JPG)
134+
- Supports frame-by-frame processing with automatic format conversion
135+
- Allows multiple client connections

‎interface/vstream/config/vstream_audio_in_config.h‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*---------------------------------------------------------------------------
2+
* Copyright (c) 2025 Arm Limited (or its affiliates).
3+
* All rights reserved.
4+
* SPDX-License-Identifier: Apache-2.0
5+
*
6+
* Licensed under the Apache License, Version 2.0 (the License); you may
7+
* not use this file except in compliance with the License.
8+
* You may obtain a copy of the License at
9+
*
10+
* www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an AS IS BASIS, WITHOUT
14+
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*---------------------------------------------------------------------------*/
18+
19+
#ifndef VSTREAM_AUDIO_IN_CONFIG_H_
20+
#define VSTREAM_AUDIO_IN_CONFIG_H_
21+
22+
//-------- <<< Use Configuration Wizard in Context Menu >>> --------------------
23+
//------ With VS Code: Open Preview for Configuration Wizard -------------------
24+
25+
// <o> Number of channels <1=>Mono <2=>Stereo
26+
// <i> Defines the number of audio channels in stream.
27+
// <i> Default: 2
28+
#ifndef AUDIO_IN_CHANNELS
29+
#define AUDIO_IN_CHANNELS 2
30+
#endif
31+
32+
// <o> Number of bits per sample <0=>8 <1=>16 <2=>24 <3=>32
33+
// <i> Defines number of bits of information in each sample.
34+
// <i> Default: 16
35+
#ifndef AUDIO_IN_SAMPLE_BITS
36+
#define AUDIO_IN_SAMPLE_BITS 16
37+
#endif
38+
39+
// <o> Sample rate <8000=>8 kHz <16000=>16 kHz <44100=>44.1 kHz <48000=>48 kHz
40+
// <i> Defines the number of samples captured per second.
41+
// <i> Default: 16000
42+
#ifndef AUDIO_IN_SAMPLE_RATE
43+
#define AUDIO_IN_SAMPLE_RATE 16000
44+
#endif
45+
46+
// <o> Streaming Device Index
47+
// <i> Defines the system index of the audio streaming device.
48+
// <i> Default: -1 (system default audio device)
49+
#ifndef AUDIO_IN_DEVICE
50+
#define AUDIO_IN_DEVICE -1
51+
#endif
52+
53+
// <s> Audio File Name
54+
// <i> Defines the name of the audio file to be used for streaming.
55+
// <i> Default: "" (use streaming device instead of file)
56+
#ifndef AUDIO_IN_FILENAME
57+
#define AUDIO_IN_FILENAME ""
58+
#endif
59+
60+
#endif

‎interface/vstream/config/vstream_audio_out_config.h‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*---------------------------------------------------------------------------
2+
* Copyright (c) 2025 Arm Limited (or its affiliates).
3+
* All rights reserved.
4+
* SPDX-License-Identifier: Apache-2.0
5+
*
6+
* Licensed under the Apache License, Version 2.0 (the License); you may
7+
* not use this file except in compliance with the License.
8+
* You may obtain a copy of the License at
9+
*
10+
* www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an AS IS BASIS, WITHOUT
14+
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*---------------------------------------------------------------------------*/
18+
19+
#ifndef VSTREAM_AUDIO_OUT_CONFIG_H_
20+
#define VSTREAM_AUDIO_OUT_CONFIG_H_
21+
22+
//-------- <<< Use Configuration Wizard in Context Menu >>> --------------------
23+
//------ With VS Code: Open Preview for Configuration Wizard -------------------
24+
25+
// <o> Number of channels <1=>Mono <2=>Stereo
26+
// <i> Defines the number of audio channels in stream.
27+
// <i> Default: 2
28+
#ifndef AUDIO_OUT_CHANNELS
29+
#define AUDIO_OUT_CHANNELS 2
30+
#endif
31+
32+
// <o> Number of bits per sample <0=>8 <1=>16 <2=>24 <3=>32
33+
// <i> Defines number of bits of information in each sample.
34+
// <i> Default: 16
35+
#ifndef AUDIO_OUT_SAMPLE_BITS
36+
#define AUDIO_OUT_SAMPLE_BITS 16
37+
#endif
38+
39+
// <o> Sample rate <8000=>8 kHz <16000=>16 kHz <44100=>44.1 kHz <48000=>48 kHz
40+
// <i> Defines the number of samples captured per second.
41+
// <i> Default: 16000
42+
#ifndef AUDIO_OUT_SAMPLE_RATE
43+
#define AUDIO_OUT_SAMPLE_RATE 16000
44+
#endif
45+
46+
// <o> Streaming Device Index
47+
// <i> Defines the system index of the audio streaming device.
48+
// <i> Default: -1 (system default audio device)
49+
#ifndef AUDIO_OUT_DEVICE
50+
#define AUDIO_OUT_DEVICE -1
51+
#endif
52+
53+
// <s> Audio File Name
54+
// <i> Defines the name of the audio file to be used for streaming.
55+
// <i> Default: "" (use streaming device instead of file)
56+
#ifndef AUDIO_OUT_FILENAME
57+
#define AUDIO_OUT_FILENAME ""
58+
#endif
59+
60+
#endif

‎interface/vstream/config/vstream_video_in_config.h‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,26 +24,47 @@
2424

2525
// <o> Frame width
2626
// <i> Defines the video stream frame width in pixels.
27+
// <i> Common frame widths: 320, 640, 800, 1024.
28+
// <i> Default: 640
2729
#ifndef VIDEO_IN_FRAME_WIDTH
2830
#define VIDEO_IN_FRAME_WIDTH 320
2931
#endif
3032

3133
// <o> Frame height
3234
// <i> Defines the video stream frame height in pixels.
35+
// <i> Common frame heights: 240, 480, 600, 768.
36+
// <i> Default: 480
3337
#ifndef VIDEO_IN_FRAME_HEIGHT
3438
#define VIDEO_IN_FRAME_HEIGHT 240
3539
#endif
3640

3741
// <o> Frame rate
3842
// <i> Defines the video stream frame rate in frames per second.
43+
// <i> Common frame rates: 15, 25, 30, 60.
44+
// <i> Default: 30
3945
#ifndef VIDEO_IN_FRAME_RATE
4046
#define VIDEO_IN_FRAME_RATE 30
4147
#endif
4248

4349
// <o> Color format <0=>Grayscale(8-bit) <1=>RGB888 <2=>BGR565 <3=>YUV420 <4=>NV12 <5=>NV21
4450
// <i> Defines the video frame color space.
51+
// <i> Default: 1
4552
#ifndef VIDEO_IN_FRAME_COLOR
4653
#define VIDEO_IN_FRAME_COLOR 1
4754
#endif
4855

56+
// <o> Streaming Device Index
57+
// <i> Defines the system index of the video streaming device.
58+
// <i> Default: -1 (system default video device)
59+
#ifndef VIDEO_IN_DEVICE
60+
#define VIDEO_IN_DEVICE -1
61+
#endif
62+
63+
// <s> Video File Name
64+
// <i> Defines the name of the video file to be used for streaming.
65+
// <i> Default: "" (use streaming device instead of file)
66+
#ifndef VIDEO_IN_FILENAME
67+
#define VIDEO_IN_FILENAME ""
68+
#endif
69+
4970
#endif

‎interface/vstream/config/vstream_video_out_config.h‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,26 +24,40 @@
2424

2525
// <o> Frame width
2626
// <i> Defines the video stream frame width in pixels.
27+
// <i> Common frame widths: 320, 640, 800, 1024.
28+
// <i> Default: 640
2729
#ifndef VIDEO_OUT_FRAME_WIDTH
2830
#define VIDEO_OUT_FRAME_WIDTH 320
2931
#endif
3032

3133
// <o> Frame height
3234
// <i> Defines the video stream frame height in pixels.
35+
// <i> Common frame heights: 240, 480, 600, 768.
36+
// <i> Default: 480
3337
#ifndef VIDEO_OUT_FRAME_HEIGHT
3438
#define VIDEO_OUT_FRAME_HEIGHT 240
3539
#endif
3640

3741
// <o> Frame rate
3842
// <i> Defines the video stream frame rate in frames per second.
43+
// <i> Common frame rates: 15, 25, 30, 60.
44+
// <i> Default: 30
3945
#ifndef VIDEO_OUT_FRAME_RATE
4046
#define VIDEO_OUT_FRAME_RATE 30
4147
#endif
4248

4349
// <o> Color format <0=>Grayscale(8-bit) <1=>RGB888 <2=>BGR565 <3=>YUV420 <4=>NV12 <5=>NV21
4450
// <i> Defines the video frame color space.
51+
// <i> Default: 1
4552
#ifndef VIDEO_OUT_FRAME_COLOR
4653
#define VIDEO_OUT_FRAME_COLOR 1
4754
#endif
4855

56+
// <s> Video File Name
57+
// <i> Defines the name of the video file to be used for streaming.
58+
// <i> Default: "" (use streaming device instead of file)
59+
#ifndef VIDEO_OUT_FILENAME
60+
#define VIDEO_OUT_FILENAME ""
61+
#endif
62+
4963
#endif

0 commit comments

Comments
 (0)