The 2000 file per directory limit - a data migration issue

Profesize posted 10th of January 2010 in Community Voice. 10 comments.

I have come across this limit now with 2 different hosting companies. Not sure how many of you are aware of this limit but as your site membership base grows then you too may like to know what is the mysterious force stopping you from transferring all of your files successfully.

Have you ever been using an FTP client (I use Filezilla) and wondered why the number of files indicated at the bottom of the remote server window never goes beyond the magical number of 1998? It's because there is a limit of 2000 files per directory and unless you talk to your hosting company to get it lifted, no matter how hard you try to transfer those files, it will continue to say 1998 files.

My question to Boonex is "Why do you not know this or if you do , then why don't you take this into consideration when setting up the data migration to move all images for example into one folder located at:


/modules/boonex/photos/data/files/

My data migration transferred over 12,000 files to the files directory. If this is such a common restriction imposed by hosting companies as the default, maybe Boonex should condsider that their Dolphineers' sites would eventually grow to a point where this default restriction would eventually become a bit of an issue?

It would of course make more sense to have more directories with less files in them.

If you have been wondering why your data migration of the photos is queueing forever then this may be why. It was for me.

Hope this helps!


Peace...Profesize (Proudly Australian too)

 
Comments
·Oldest
·Top
Please login to post a comment.
AlexT
The limit you are experienced is limit of your memory limit in php, or it is timeout on connection to mysql.. or something not related to file number limit. Try to increase memory limit for php along with connection timeout for mysql.

There is no such limit on filesystem (about 2K file) ! the most close limit is about 30K subfolders(not files), but Dolphin do not use subfolders in such cases!
Profesize
Well my last two host companies changed this level from 2000 to 20000 for me and when they increased it, the problem was resolved.......so it must exist.
mydatery
Thus this shows that the issue is with your host and NOT with Dolphin. Note that your host changed it. The vast majority of hosts will not alter the Dolphin script, especially without gettting paid to alter it.
Profesize
At no point did I say that the hosting companies ever make a change to the Dolphin script. It was a setting on the server they changed. NOT THE DOLPHIN SCRIPT.

On both occasions when this limit of 2000 was increased it fixed my problem.
mydatery
And your asking Boonex why they haven't taken this into consideration. How is Boonex supposed to take into consideration a limit that your host is placing on the server? This question has nothing to do with Boonex and they can not in any shape, manner or form do anything about it for you.

What I'm really wondering is if your actually comprehending what your writing? Look at your OP:

"My question to Boonex is "Why do you not know this or if you do , then why don't you take this see more into consideration when setting up the data migration to move all images for example into one folder located at:"

This is NOT an issue that has to do with installs. Are you suggesting that Boonex should add another check line to make sure your server is set up to accept a certain number of files before you do your migration? Perhaps they should also run a check to make sure your getting the best deal on your internet service before you do the migration.
Profesize
No need to be ridiculous. All I am saying is that if I have had this issue with two different host companies then it is more than likely that other members of Unity will have the same issue when their sites reach the amount of members that will cause certain directories to exceed this 2000 file limit.

Do you happen to have any directories with more than 2000 files in them? Just curious.
DosDawg
it is a standard default on most OS's that are released, really has nothing to do with a hosting company setting limitations, and has nothing to do with boonex. this would be a good sticky note for a migrations thread though.

there are other limitations that are on most all *nix distros, and until you hit that limit, you really dont even realize they are there. these limitations of bottom boundaries are just part of the programming of the OS.

MD is correct, most hosts do not alter the server see more settings, that is one of the reasons that boonex recommends using a VPS (minimum) or a Dedicated Server. There are hosts who will run an optimized shared environment for hosting dolphin (no names named), and those configurations are outside the standardized setup that you would get from an ISO image.

Summary:
most everything that is default and mass released has limitations, there are upper and lower boundaries that are considered by these developers in all cases, and how they arrive at their calcs are most likely proprietary to that OS build. If these boundary limitations are of a concern, then by all means grab yourself a dedicated server, then you can fight the fight, and when the whole server comes crashing down, you will have a better understanding of what is going on. (maybe).

Regards,
DosDawg
DosDawg
AlexT,
just want to clarify something with you in regards to your uppermost post:

"There is no such limit on filesystem (about 2K file) ! the most close limit is about 30K subfolders(not files), but Dolphin do not use subfolders in such cases!"

maybe its just a misunderstanding, but you mention the filesystem limitation and you state that dolphin do not use subfolders in such cases. was this referring to the migration itself or overall in general?
blaine69
This is not a BOONEX problem ONLY, I had this same situation with my online store (x-cart) that has OVER 4,000 product images.... when I questioned my Service Provider (Arvixe), they told me that all the files are there, but only the first 1998 are viewable.... they INCREASED the limit for me and now ALL MY FILES are visible.

Blaine.
blaine69
...... just make sure your Service Provider will/can increase the limit for you before you sign on.

Blaine.
 
 
Below is the legacy version of the Boonex site, maintained for Dolphin.Pro 7.x support.
The new Dolphin solution is powered by UNA Community Management System.
PET:0.05425500869751