Abstract Body (Do not enter title and authors here): Background: Toll-like receptor 3 plays a role in the development of calcific aortic valve disease (CAVD). However, there are currently no pharmacological treatments for CAVD, and the discovery of de novo molecules targeting a specific protein is a time-intensive and financially demanding process. It is hypothesized that language models, trained on encodings of molecular graphs, can tailor the design of novel molecules with specific molecular and pharmacological properties.

Objective: This study aims to develop a chemical language model tailored for designing novel molecules that target the inhibition of TLR3.

Methods: A language model was pre-trained on 25,000 chemical structures sourced from the ChEMBL database. Each chemical structure was represented through Simplified Molecular Input Line Entry System (SMILES) strings, which were subsequently tokenized into discrete atomic and functional group tokens. The model leverages an Average-Stochastic Gradient Descent Weight-Dropped Long Short-Term Memory (AWD-LSTM) architecture. Transfer learning was employed for domain-specific fine-tuning of the pre-trained language model on the target chemical structures.

Results: The model achieved strong performance by unfreezing the pre-trained language model’s parameters for fine-tuning in all layers. Specifically, it achieved an accuracy of 85.49%, a weighted F1-score of 83.28%, and a perplexity of 1.44. This approach yielded better results compared to partially freezing the model’s parameters, where only the latter layers were fine-tuned. The partial freezing approach resulted in an accuracy of 85.06%, a weighted F1-score of 82.44%, and a perplexity of 1.44.

Conclusion: A chemical language model was developed for designing novel molecules that target the inhibition of TLR3. This provides an efficient and automated method for developing novel molecules with specific molecular and pharmacological properties.