CAT-TPT: Class-Agnostic Text-based Test-time Prompt Tuning for Vision-Language Models