Similarity Microservices with Automated Tag Generation

Abstract

A scalable web-based application addresses the problem of identifying contextually similar text posts and automatically generating relevant tags using Natural Language Processing (NLP) techniques, integrated within a modular and microservices architecture. The system enables users to create and manage text-based posts. Upon submission, two critical backend services are triggered: the Similarity Engine and the Tag Generator. The Similarity Engine converts post content into sentence embeddings using models like Sentence Transformer, and uses cosine similarity to compare the new post with all existing posts. A similarity threshold of 75% to 85% is applied. If the similarity score exceeds  this threshold, the matched posts are recommended to the user; otherwise, the new post is added. This allows the application to recommend semantically related posts, even if they use entirely different wording.

Simultaneously, the Tag Generator microservice uses the RAKE algorithm to extract the most important keywords from the post content, which are then used as tags for categorization, filtering, and search optimization. This process is entirely automated and does not rely on manually assigned labels. This approach demonstrates effective semantic text processing and modern software engineering practices, making it adaptable for applications such as search engines, recommendation systems, e-learning platforms, and customer support tools.