neuromorphic_drivers is a library to interact with USB Neuromorphic devices. The drivers were written from scratch with portability and performance in mind.
Supported devices and features
Name | Type | Resolution | Data types | Mask | Synchronize | Rate limiter | Temperature | Illuminance |
---|---|---|---|---|---|---|---|---|
Prophesee EVK4 | Camera | 1280 × 720 | DVS, trigger | ✓ | ✓ | ✓ | ✓ | ✓ |
Prophesee EVK3 HD | Camera | 1280 × 720 | DVS, trigger | ✓ | - | ✓ | - | - |
This table lists fratures supported by this library. Some devices support unlisted features or features marked as "no" that have yet to be added to neuromorphic_drivers.
Name | Links |
---|---|
Prophesee EVK4 | https://www.prophesee.ai/event-camera-evk4/ |
Prophesee EVK3 HD | https://www.prophesee.ai/event-based-evk-3/ |
Python
Get started
On Linux, run the following comman after installing the package to install UDEV rules.
The following script reads data from a connected device.
# print a table that lists connected devices
# packet["dvs_events"] is a structured numpy array
# with dtype [("t", "<u8"), ("x", "<u2"), ("y", "<u2"), ("on", "?")])
pass
# packet["trigger_events"] is a structured numpy array
# with dtype [("t", "<u8"), ("id, "<u1"), ("rising", "?")])
pass
The keys in a packet depend on the device and are only present if the associated array is not empty. Packets contain variable numbers of events (typically a few thousand to a few hundred thousand) covering a variable amount of time (typically a few dozen microseconds to a few milliseconds).
Device configuration
The configuration, specific to each device, defines all the device's parameters – from biases to region of interest. It can be specified when opening a device and it can be updated at any time. The default configuration for device d
is defined in python/neuromorphic_drivers/generated/devices/d.py.
=
...
-= 10
-= 10
nd.open
may open any supported device. However, it only considers matching devices if a configuration is specified. Passing the wrong configuration type to update_configuration
raises an error.
Rate limiter
Prophesee's EVK4 has a hardware event-rate limiter that randomly drops events when the sensor generates more than a given number per second. The limiter has two parameters. reference_period_us
defines a count period in µs (maximum 200). maximum_events_per_period
defines the number of events per period beyond which the limiter activates. The effective limit is maximum_events_per_period / reference_period_us
in events/µs.
=
...
Raw mode
Converting the raw USB data into events can be an expensive operation if the data rate is high. Raw mode skips parsing and can be useful if one simply wishes to store the data into a file to be processed offline.
# in raw mode, packet is a "bytes" object
Other open options
...
The fields of the USB configuration are identical across devices, but the default values depend on the device. The values below are for a Prophesee EVK4.
: = 131072 # size of each buffer in the ring, in bytes
: = 4096 # number of buffers in the ring, the total size is ring_length * buffer_length
: = 32 # number of libusb transfers submitted in parallel
: = False # whether to enable Direct Memory Access
More examples
See python/examples for different usage examples.
python/examples/any_display.py implements a live event viewer with exponential decays caculated by the GPU. It requires vispy and glfw (pip install vispy glfw pyopengl
).
python/examples/evk4_plot_hot_pixels generates plots and require Plotly (pip install plotly pandas kaleido
).
Contribute
Local build (first run).
Local build (subsequent runs).
Before pushing new code, run the following to lint and format it.
; ;
The files in python/python/neuromorphic_drivers/generated are generated by the build script (python/build.rs) and should not be modified directly. They are included in this repository for documentation purposes.
When making changes to the Rust library and the Python wrappers at the same time, change neuromorphic-drivers = "x.y"
to neuromorphic-drivers = {path = "../drivers"}
to use the most recent version of the Rust code.
Rust
Documentation
See https://docs.rs/neuromorphic-drivers/latest/neuromorphic_drivers/ for documentation.
UDEV rules
-
Write the following content to /etc/udev/rules.d/65-neuromorphic-drivers.rules.
SUBSYSTEM=="usb", ATTRS{idVendor}=="152a",ATTRS{idProduct}=="84[0-1]?", MODE="0666" SUBSYSTEM=="usb", ATTRS{idVendor}=="04b4",ATTRS{idProduct}=="00f[4-5]", MODE="0666"
-
Run the following commands (or reboot the machine).
Contribute
Run a specific test.
Publish on crates.io.
Performance
Event rate
Recent event-based cameras, such as the Prophesee EVK4, can generate more data than can be processed in real-time under certain circumstances. While the data can always be moved to the computer's memory in real-time, the simplest algorithms (including converting the raw USB bytes to a {t, x, y, polarity}
event representation) struggle to keep up during data rate peaks. This library uses seperate threads for reading (USB to memory) and processing (memory to memory or disk) with a circular buffer (ring) at the interface. Short data bursts are seemlessly absorbed by the ring and are typically not an issue, even though they brifely cause a spike in latency. However, persistent high data rates cause the ring to slowly fill and cause the program to eventually crash. Depending on the use-case, one of the following work arounds can be applied:
- reduce the camera's sensitivity (usually by changing
diff_off
anddiff_on
) - enable the event rate limiter if the device supports it (the limiter randomly drops events before sensing them to the computer, reducing bandwidth issues but significantly degrading the quality of transient bursts)
- reduce the camera's spatial resolution by masking rows and columns
- change the environment if possible (avoid flickering lights, reduce the optical flow, remove background clutter, keep large and fast objects out of the field of view)
- call
device.clear_backlog(until=0)
whenever thebacklog
becomes too large (the maximum backlog is the size of the ring minus the size of the transfer queue) - use
nd.open(raw=true)
to skip the parser and directly access the USB bytes, typically to save them to a file
Direct Memory Access
This library relies on libusb for all USB communications. libusb supports Direct Memory Access (Linux only for now, though other platforms may be added in the future), which allows the USB controller to directly write packets to memory without requiring CPU (and OS kernel) intervention. While this can increase performance and reduce CPU usage, DMA comes without caveats that we disabled it by default. Users when knoweldge of their USB controller and its limitations may want to enable it to increase performance on embedded systems.
USB drivers have a limited number of DMA file objects (128 / 256 / 512 / 1024). This is typically not enough to accomodate event bursts (we use 4096 buffers by default for the EVK4, with 131072 bytes per buffer). The code fall backs to non-DMA buffers if all the DMA buffers are used (for instance the first 128 buffers would be DMA and the rest would be non-DMA) and may result in variable performance over time.
Using all the available DMA buffers can cause other USB transfers to fail (including control transfers to configure the device).
Using DMA thus requires one of the following workarounds:
- Use a small number of DMA buffers and copy packets to a larger ring before processing (this somewhat defeats the purpose of DMA and increases memory copies).
- Use larger buffers (around 1 MB) to inccrease the ring size without increasing the number of buffers, at the cost of increased latency.
- Ensure that processing can always keep up with the data rate (sparse scenes / low-sensitivity biases / event-rate limiter / simple processing). Note that simply expanding vectorized EVT3 events to 13-bytes DVS events is not real-time during data peaks.