Widespread availability of public digitized text has greatly facilitated the advancement of natural language processing (NLP), leading in turn to public-interfacing models like ChatGPT. Text processing could also be extremely valuable for processing high-stakes private data, like social workers’ notes, healthcare records, or customer data. Information extraction could help a social worker quickly search hundreds of notes for a caretaker’s contact information (Gandhi et al., 2022), and initiatives like Amazon Comprehend Medical evidence the demand for NLP in healthcare settings. However, safety risks and the need for responsible data practices hinder the development and deployment of models in these domains. Models are prone to absorbing and amplifying data biases, and potential unfairness makes models unusable in high-stakes domains. Furthermore, text data is extremely difficult to fully anonymize. Poorly-anonymized high-stakes data cannot be shared annotators or external researchers, nor can it be used it to train deployed models, as generative models are prone to outputting sensitive information from training data (Carlini et al., 2023). This proposal aims to address these challenges by developing text generation tools to create realistic synthetic data that can facilitate research and model development while improving model fairness and minimizing privacy violations. The core technical contribution includes the development of constraints for controllable text generation that balance the trade-off between removing private or biased information and ensuring synthetic text is realistic enough to be useful. To facilitate developing models for real private data, we will draw on the PI’s ongoing partnership with the Allegheny County Department of Human Services to examine notes about child welfare cases (Gandhi et al., 2022; Field et al., 2023) and will also draw from existing medical NLP research (Johnson et al., 2016; Uzuner et al., 2011; Gandhi et al., 2021). This work is critical for building text processing systems in high-risk settings with private data. The ability to synthesize and anonymize data would enable products designed over data internal to Amazon, such as public-facing dialog agents for interacting with customers, as well as products designed for external private data, such as expansions of Amazon Comprehend Medical or new tools for processing social workers’ notes.

On-device speech processing is getting more attention as machine learning algorithms continue improving. Understanding how big a model needs to be is still an open question. Diarization is not an exception to this trend; the current state-of-the-art systems can get low diarization error rates using these large-scale self-supervised learning (SSL) models. However, these systems are rarely deployable because they are inefficient, computationally expensive and need special technical requirements. Model compression and subsampling techniques have shown to be suitable solutions to solve the problems of these models to some extent. Although the same methods can be used for various downstream tasks (including automatic speech recognition, speaker identification, emotion recognition, etc.), we aim to focus on diarization, as it is usually in the first stages of any speech-processing application. In this proposal, we will study how to build efficient diarization models based on self-supervised models that can be deployed on-device. The core pipeline uses a state-of-the-art End-to-end neural diariazition system (EEND-EDA), in which a SSL model replaces the encoder. As we want to focus on the impact of the self-supervised models, this proposal considers two avenues: compressing and subsampling.

On the one hand, we propose investigating successful compressing methods such as weight pruning, head pruning, low-rank approximation and knowledge distillation. We will jointly optimize the self-supervised and diarization losses to achieve our purpose. Our starting point is a WavLM self-supervised model combined with EEND-EDA. The WavLM will act as the encoder of the EEND-EDA model, whereas the rest of the EEND-EDA modules will become a back-end. For the pruning options and the low-rank approximation, our approach will trim a set of weights and optimize the whole system until convergence. For the distillation, inspired by DistilHuBERT, we will select specific layers from the Transformer and compute prediction heads optimized for diarization. On the other hand, we will study the effect of shortening the sequences, i.e., subsampling along the time axis in the self-supervised learning models. We follow this strategy as it has been shown that the subsampling can improve the performance and speed up the inference. Two main approaches will be explored: variable-length (for example, pooling variable-length frame representations into segment representations) and fixed-length (for example, removing odd frames) subsampling. We will use our crafted distilled version of WavLM as the encoder of the EEND-EDA model as our starting point. Using a student-teacher approach, the student will minimize the distance between the sequence of teacher representations and a projection (prediction heads) of its own representations. The prediction heads are connected to the EEND-EDA and optimized accordingly.

The combination of the compressing methods with the subsampling techniques will provide insights into how aggressive the subsampling and the compression could be and where the boundaries are. Finally, to document the performance, this study will use diarization error rate (DER), real-time factor, multiply-accumulate operations, the number of parameters, and wall clock time as metrics. We plan to test our algorithms on the Callhome, AMI, and Dihard III datasets.