| Title: |
Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian |
| Authors: |
Auriemma, Serena, Miliani, Martina, Madeddu, Mauro, Bondielli, Alessandro, Passaro, Lucia, Lenci, Alessandro |
| Publication Year: |
2024 |
| Collection: |
Computer Science |
| Subject Terms: |
Computer Science - Computation and Language, Computer Science - Artificial Intelligence, 68T50, 68T07, I.2.7 |
| More Details: |
Addressing the challenge of limited annotated data in specialized fields and low-resource languages is crucial for the effective use of Language Models (LMs). While most Large Language Models (LLMs) are trained on general-purpose English corpora, there is a notable gap in models specifically tailored for Italian, particularly for technical and bureaucratic jargon. This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in these specialized contexts. Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models. We evaluated the models on downstream tasks such as document classification and entity typing and conducted intrinsic evaluations using Pseudo-Log-Likelihood. The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting. Furthermore, the application of calibration techniques and in-domain verbalizers significantly enhances the efficacy of encoder models. These domain-specialized models prove to be particularly advantageous in scenarios where in-domain resources or expertise are scarce. In conclusion, our findings offer new insights into the use of Italian models in specialized contexts, which may have a significant impact on both research and industrial applications in the digital transformation era.
Comment: Submitted to 'Language Resource and Evaluation' |
| Document Type: |
Working Paper |
| Access URL: |
http://arxiv.org/abs/2407.20654 |
| Accession Number: |
edsarx.2407.20654 |
| Database: |
arXiv |
