On-device speech processing is getting more attention as machine learning algorithms continue improving. Understanding how big a model needs to be is still an open question. Diarization is not an exception to this trend; the current state-of-the-art systems can get low diarization error rates using these large-scale self-supervised learning (SSL) models. However, these systems are rarely deployable because they are inefficient, computationally expensive and need special technical requirements. Model compression and subsampling techniques have shown to be suitable solutions to solve the problems of these models to some extent. Although the same methods can be used for various downstream tasks (including automatic speech recognition, speaker identification, emotion recognition, etc.), we aim to focus on diarization, as it is usually in the first stages of any speech-processing application. In this proposal, we will study how to build efficient diarization models based on self-supervised models that can be deployed on-device. The core pipeline uses a state-of-the-art End-to-end neural diariazition system (EEND-EDA), in which a SSL model replaces the encoder. As we want to focus on the impact of the self-supervised models, this proposal considers two avenues: compressing and subsampling.

On the one hand, we propose investigating successful compressing methods such as weight pruning, head pruning, low-rank approximation and knowledge distillation. We will jointly optimize the self-supervised and diarization losses to achieve our purpose. Our starting point is a WavLM self-supervised model combined with EEND-EDA. The WavLM will act as the encoder of the EEND-EDA model, whereas the rest of the EEND-EDA modules will become a back-end. For the pruning options and the low-rank approximation, our approach will trim a set of weights and optimize the whole system until convergence. For the distillation, inspired by DistilHuBERT, we will select specific layers from the Transformer and compute prediction heads optimized for diarization. On the other hand, we will study the effect of shortening the sequences, i.e., subsampling along the time axis in the self-supervised learning models. We follow this strategy as it has been shown that the subsampling can improve the performance and speed up the inference. Two main approaches will be explored: variable-length (for example, pooling variable-length frame representations into segment representations) and fixed-length (for example, removing odd frames) subsampling. We will use our crafted distilled version of WavLM as the encoder of the EEND-EDA model as our starting point. Using a student-teacher approach, the student will minimize the distance between the sequence of teacher representations and a projection (prediction heads) of its own representations. The prediction heads are connected to the EEND-EDA and optimized accordingly.

The combination of the compressing methods with the subsampling techniques will provide insights into how aggressive the subsampling and the compression could be and where the boundaries are. Finally, to document the performance, this study will use diarization error rate (DER), real-time factor, multiply-accumulate operations, the number of parameters, and wall clock time as metrics. We plan to test our algorithms on the Callhome, AMI, and Dihard III datasets.