The limit you are experienced is limit of your memory limit in php, or it is timeout on connection to mysql.. or something not related to file number limit. Try to increase memory limit for php along with connection timeout for mysql.

There is no such limit on filesystem (about 2K file) ! the most close limit is about 30K subfolders(not files), but Dolphin do not use subfolders in such cases!