SketchWhisper

Abstract

This paper presents a pioneering method for developing a system that transforms spoken input into vector sketch-style images, aiming to bridge the semantic divide between natural language descriptions and visual representations. Our approach introduces novel methodologies that diverge from conventional paradigms.Fundamentally, we employ stable diffusion models, customized and finely tuned on specialized datasets to optimize vector sketch generation. This innovative method directly tackles the challenge of maintaining coherence and stability within the generated sketches, ensuring fluid transitions and accurate depiction of stroke patterns. Moreover, we investigate the fusion of dual transformer models, with one dedicated to processing textual input and the other specialized in synthesizing vector stroke data for images. This integration aims to synergistically combine the strengths of transformer architectures with state-of-the-art techniques in vector sketch generation, thereby augmenting the system's capacity to capture intricate linguistic semantics and translate them into visually cohesive sketches. Our experimental and evaluative endeavors underscore the effectiveness and adaptability of these methodologies. Envisioned as a transformative tool, our system holds wide-ranging applications encompassing digital art creation, educational support, and assistive technologies. This research significantly propels the evolution of speech-to-draw systems, fostering a more intuitive and expressive landscape for human-computer interaction.