AI at Meta, a subsidiary of Meta, announced the open-source Perception Encoder Audiovisual (PE-AV) and positioned it as a key technology engine to drive SAM Audio to achieve cutting-edge audio separation effects. Based on the earlier Perception Encoder system, PE-AV natively integrates audio and visual perception to align audio, video (or audio and video clips) and text information in the same representation space.
According to public information, PE-AV focuses on multi-modal capabilities, which can be used for sound retrieval, sound detection, and richer audio and video scene understanding, and has achieved leading performance in a number of audio and video benchmarks. In terms of supporting resources, the code is open source on GitHub; Model weights are published on Hugging Face at multiple scales, making it easy for developers to reuse them in audio separation, cross-modal retrieval, and understanding tasks. It should be noted that the specific comparison settings and reproducible details of the official "benchmark leading" conclusion are still subject to the disclosure of papers and model cards.
FAQ
Q: What is the technical component of PE-AV?
A: PE-AV is an audio-visual multimodal perception encoder used to map audio, video, and text into a unified embedding space to support downstream separation and understanding tasks.
Q: What is the relationship between PE-AV and SAM Audio?
A: PE-AV is described as an important technology engine for SAM Audio to achieve cutting-edge audio separation effects, which can be used to improve audio-visual alignment and separation-related capabilities.
Q: What daily or application scenarios can PE-AV do?
A: PE-AV can be used for sound detection, cross-modal retrieval (finding sound from text or images), and more complete audio and video scene understanding and analysis.
Q: Where can I get the code and model of PE-AV?
A: The code is publicly available in GitHub's facebookresearch/perception_models repository, and model weights are available in multiple scale versions under Hugging Face's facebook organization.
Q: What are the limitations I need to be aware of when using PE-AV?
A: Different weights and datasets may have their own licenses and terms of use, and some models may need to apply for access on the platform side.
