PE-AV (Perception Encoder Audiovisual) Open Source Interpretation: The audio separation engine that powers SAM Audio

1. Abstract

PE-AV (Perception Encoder Audiovisual) is Meta's open-source audio-visual joint encoder family, which adds native audio capabilities on the basis of Perception Encoder to align video, audio, audio and video and text representation with a unified embedding space. It is used to underpin key components of SAM Audio and leads several audio/video retrieval and understanding benchmarks.

1. Core features

1. Multi-modal unified embedding: It supports feature encoding and similarity calculation of audio, video, audio-video, and text at the same time.
2. Audio and video retrieval and alignment: It can do cross-modal search such as "text to find sound/picture" and "screen to find sound".
3. Multi-size/multi-version weights: Small/base/large, as well as configurations such as "16-frame" and "all frames", are available to facilitate trade-offs between effects and computing power.
4. Engineering reusability: Released in the same warehouse as the perception_models ecosystem, which is convenient for collaboration with PE series and downstream multi-modal applications.

1. Installation

1. Clone code: git clone https://github.com/facebookresearch/perception_models
2. Create an environment and install dependencies: install according to the repository requirements.txt / setup.py instructions (dependencies may vary for different platforms).
3. Get weights: When running the example, it will pull the corresponding checkpoint from Hugging Face (such as pe-av-large, etc.).

1. Typical use cases

1. Audio and video search: Use one sentence to find "dialogue clips with sirens" from the video library.
2. Sound cue assisted understanding: Combine the picture to improve sound source recognition and scene description in noisy scenes.
3. Upstream perception of audio separation/editing: Provides stronger audio-video alignment representation for interactive separation like SAM Audio.
4. Data annotation and quality inspection: use cross-modal similarity to screen samples with "inconsistent picture and sound".

1. Ecology and Competing Products

1. Ecology: PE-AV is a perception_models of Meta; Model weights are published as collections in Hugging Face for easy reproducibility and integration.
2. Competing product ideas: Compared with audio encoders or video encoders only, PE-AV focuses on the unified space of "audio-visual correspondence learning"; Compared with the CLIP method, it is extended to the joint characterization of audio and audio and video, which is closer to the real video task.

1. Limitations and precautions

1. Computing power and throughput: The configuration of video frame rate will significantly affect the video memory and speed, and you need to choose small/base/large and frame strategy according to the service.
2. Data domain shift: In specific languages, specific audio types, or strong noise conditions, cross-modal alignment may decline, so small-scale verification is recommended.
3. Copyright and privacy: When processing public video/audio conversations with people, data compliance and authorization requirements must be observed.

1. Project address

https://github.com/facebookresearch/perception_models

1. FAQ

(PE-AV open source model) What input modalities are supported?

Answers

support audio, video, audio, video, and text, and output embedded representations that can be used for similarity calculations.

Question (PE-AV installation) Where are the weights downloaded from, and do they need to be placed manually?

Answers

are usually automatically pulled by Hugging Face to pull the corresponding checkpoint; For offline environments, you need to manually download and configure the path according to the repository instructions.

Question (PE-AV vs. SAM Audio) What is the relationship between the two?

Answer

PE-AV is the perception/encoding engine that drives several core components of SAM Audio, providing enhanced audio-visual alignment.

Question (PE-AV checkpoint) Should I choose pe-av-small/base/large or 16-frame?

The larger the answer

, the stronger the effect but more computing power; 16-frames are more resource-saving, and all frames make better use of video information, so it is recommended to conduct comparative experiments based on service speed/cost constraints.