There are way too many variables you need to consider to make a simple determination of what could be causing your issues. The RAY RMS Server is based upon a beta of Red5 which is still undergoing development, your server setup (memory, cpu, bandwidth) all contribute to these types of problems. Red5 has been shown to reliably work with several hundred simultaneous connections. A VPS is running not only your site, but potentially many others at the same time and all of them are sharing resources, see more so a heavily used site on the same VPS as yours would probably result in a performance drop across all sites on it. You are sharing bandwidth as well. Other features you may be running on your site could also contribute to the issue. I'm not an expert on this, and welcome corrections to this post, but in my experience, a dedicated server with enough memory, at least dual CPU's and as fast a port speed as you can afford would probably solve your issues unless it's a piece of software you are running that may be causing the problems. When I finally settled on a solution for my site, I anticipated growth for several years into the future and planned accordingly. I added as much memory as I could afford to, settled on a multi-cpu setup to share load, and used different servers for the main site and chat area. I also use a 3rd party chat application running on a Macromedia Flash Server. In tests, the server could reliably handle up to almost 5000 simultaneous connections in chat before hitting the 60% cpu utilization. With 50 or so in chat, the cpu usage remains flat at about 2% and memory usage is minimal. My situation probably differs extensively from yours, but I thought I'd add my experiences if it helps at all.
Below is the legacy version of the Boonex site, maintained for Dolphin.Pro 7.x support.
The new Dolphin solution is powered by UNA Community Management System.