Open Access Open Access  Restricted Access Subscription Access

AI POWERED OS ASSISTANT

Mr.Midhun M, Prashannah A, Praveena P, Prathyush Ram GS, Rithul Janardhanan

Abstract


The AI Powered OS Assistant is a desktop automation system that allows users to control computer operations using voice and text commands. It uses speech recognition, NLP, LLMs, and Python automation to understand commands and perform tasks such as application control, file management, and system monitoring. A confirmation mechanism is used for high risk operations to improve safety and user control. SQLite is used to store command history and system settings. The system aims to reduce manual effort and provide a simple and intelligent way to interact with the operating system.


Full Text:

PDF

References


Radford, J. W. Kim, T. Xu, et al., “Robust Speech Recognition via Large-Scale Weak Supervision,” Proceedings of the 40th International Conference on Machine Learning, vol. 202, pp. 28492–28518, 2023. (Google Research)

Vaswani, N. Shazeer, N. Parmar, et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems, vol. 30, 2017.

J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proceedings of NAACL-HLT, pp. 4171–4186, 2019. (ACL Anthology)

S. Yao, J. Zhao, D. Yu, et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” International Conference on Learning Representations, 2023.

T. Schick, J. Dwivedi-Yu, R. Dessì, et al., “Toolformer: Language Models Can Teach Themselves to Use Tools,” arXiv preprint arXiv:2302.04761, 2023. (arXiv)

Q. Wu, G. Bansal, J. Zhang, et al., “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation,” arXiv preprint arXiv:2308.08155, 2023. (arXiv)

M. Li, Y. Zhao, B. Yu, et al., “API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs,” Proceedings of EMNLP, pp. 3102–3116, 2023. (ACL Anthology)

K. Cheng, Q. Sun, Y. Chu, et al., “SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents,” Proceedings of ACL, pp. 9313–9332, 2024. (ACL Anthology)

W. Hong, W. Wang, Q. Lv, et al., “CogAgent: A Visual Language Model for GUI Agents,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281–14290, 2024. (Open Access CVF)

T. Xie, D. Zhang, J. Chen, et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments,” arXiv preprint arXiv:2404.07972, 2024.


Refbacks

  • There are currently no refbacks.