Check the server logs for stats on the URL for the RSS feed, you will see that almost no one uses it and the hits are repeated from the same IPs, so it does not affect as many users as you see hits.

RSS feeds are automated to reindex based upon it's config settings (measured in seconds) to refresh itself. That eats up the CPU, database queries, and memory allocations for the indexing process. You will notice a significant difference.