ffmpeg processes limit

I the Flash setting for the video module, there is a setting for how many processes are run.

 

Yet in SSH I see this,

 

results,

 

flash setting.JPG · 15.4K · 1318 views
ffmpeg processes.JPG · 27K · 1434 views
load.JPG · 11.7K · 1282 views
ManOfTeal.COM a Proud UNA site, six years running strong!
Quote · 6 Jun 2014

Personally, I always thought 'Maximum number of processing files' meant the number of video file you wish to allow your web server to process at one time before moving on to other video files needing to be processed????

IDK

Quote · 6 Jun 2014

I thought that setting would determined how many files would move from pending to processing but when checking in the back end, only one file would move from pending to processing.  The question is if ffmpeg can process multiple files simultaneously and if so, does it save on processing time if you have the resources?  If it takes the same amount of time to process one file, load the next, process, load the next, process, as it does to load three files and process then it really does not matter if we process one at a time or three at a time.

Geeks, making the world a better place
Quote · 7 Jun 2014

Well, no matter the flash setting, the system will try to process all that are sent to it.

I tested by uploading 20 videos, I had as many as 10 processes running. The server load got so high, most failed during processing.

Mind you this was my test server, with nothing else going..

server1specs.JPG · 56.1K · 1333 views
ManOfTeal.COM a Proud UNA site, six years running strong!
Quote · 7 Jun 2014

Actually "Maximum number of processing files" setting means that it will process up to 2 files per one run.

Dolphin runs video processing every 5 minutes (via cron jobs).

The problem maybe with big video files, if they are processing longer than 5 minutes:

for example you uploaded 5 big video files and processing of one file takes approximately 15 minutes, and cron runs every 5 minutes:

- after 1st cron run you will 2 files in processing and 3 files are pending processing

- after 2nd cron run (after 5 minutes) you will have 4 files processing and 1 file pending processing

- after 3rd cron run (after 10 minutes) you will have 5 files processing and 0 files pending

- then after 15 minutes you will have 3 files processing

- after 20 minutes 1 file processing

- after 25 minutes all video files should be ready

To limit the server load you can set to run video processing every 30 minutes (in table sys_cron_jobs)

Rules → http://www.boonex.com/terms
Quote · 9 Jun 2014

 

To limit the server load you can set to run video processing every 30 minutes (in table sys_cron_jobs)

 That's what I will do then; thanks.

ManOfTeal.COM a Proud UNA site, six years running strong!
Quote · 9 Jun 2014

Would it be "smarter" for dolphin to not process another file until after there is a free processing slot open?
For example: (5 videos to process)

after 1st cron run you will have 2 files in processing and 3 files are pending processing
after 2nd cron run (after 5 minutes) you will have 4 files processing and 1 file pending processing 

  • If 2 files are still processing, then it does nothing.
  • If 1 files are still processing, then it begins processing another file
  • If 0 files are processing, then it loads 2 more files to process 

Then reduce the cron jobs to check/run every minute.  This is so that users do not have to wait until half an hour later (or even longer) to see their videos, and can see them instantly after they are done processing, which should be fairly quick (on a properly configured server)
This should also reduce the load on a server because more than 2 videos would never be processing, and how ever many cores you have can tackle processing just only 2 files, instead of trying to process 5 (or potentially more) videos.  

Ultra Newb reporting for duty.
Quote · 9 Jun 2014

 

Would it be "smarter" for dolphin to not process another file until after there is a free processing slot open?
For example: (5 videos to process)

after 1st cron run you will have 2 files in processing and 3 files are pending processing
after 2nd cron run (after 5 minutes) you will have 4 files processing and 1 file pending processing 

  • If 2 files are still processing, then it does nothing.
  • If 1 files are still processing, then it begins processing another file
  • If 0 files are processing, then it loads 2 more files to process 

 

Then reduce the cron jobs to check/run every minute.  This is so that users do not have to wait until half an hour later (or even longer) to see their videos, and can see them instantly after they are done processing, which should be fairly quick (on a properly configured server)
This should also reduce the load on a server because more than 2 videos would never be processing, and how ever many cores you have can tackle processing just only 2 files, instead of trying to process 5 (or potentially more) videos.  

 so what if the first two videos are long videos. With your suggestion, users will still have to wait a significant amount of time. At least how it is now, if the first two videos are long then other videos will still get processed and more persons are happy instead of having to wait. And God forbid your video is the last one uploaded. You may be waiting for hours depending on the size of the videos before yours. It is like going to the store and there are only 2 registers open with 20 people waiting. The way it is now is like opening 2 new registers every 5 minutes thus allowing everyone to get through faster

caredesign.net
Quote · 10 Jun 2014

 

With your suggestion, users will still have to wait a significant amount of time.

 I agree, it's not going to work this way.

I'm going to have to do something, I mean really, my site is busy with videos.

Big wake up for me, Dolphin is not meant for a "big, active site," apparently.

ManOfTeal.COM a Proud UNA site, six years running strong!
Quote · 10 Jun 2014

 That's a good point, I didn't think about that the first 2 videos could be huge files.  But with saying that, my logic was that:

If a 4 core server is being used, and lets say there are 5 (big) videos being uploaded then:
Those 4 cores would process 2 videos much quicker than having 4 cores processing 5 videos.  Even if it takes 10 minutes to process a (big) video, then the first 2 videos would be processed in approx 20 minutes, versus 5 videos that will take 50 minutes (or longer because more videos being processed will add to the wait time).
 

But if my theory is wrong (which Im putting on my "to-test" list) then what would probably be a better way of going about processing them? (outside of having a beefy server) 

 

so what if the first two videos are long videos. With your suggestion, users will still have to wait a significant amount of time. At least how it is now, if the first two videos are long then other videos will still get processed and more persons are happy instead of having to wait. And God forbid your video is the last one uploaded. You may be waiting for hours depending on the size of the videos before yours. It is like going to the store and there are only 2 registers open with 20 people waiting. The way it is now is like opening 2 new registers every 5 minutes thus allowing everyone to get through faster

 

Ultra Newb reporting for duty.
Quote · 10 Jun 2014

 Oh no Newt, please dont begin crushing my dreams, lol

 Big wake up for me, Dolphin is not meant for a "big, active site," apparently.

 

Ultra Newb reporting for duty.
Quote · 10 Jun 2014

The number of threads set is suppose to help with speeding up processing; Dolphin by default uses one thread.  I tried setting more than one thread in a test but did not seem to improve the speed; could be that I did not set it properly.  I checked and Boonex appears to have compiled ffmpeg to use more than one thread; check host tools on this.

I found the following on a site discussion:

FWIW I ran some x.264/ffmpeg encoding tests on a Dual Xeon 5620 (16 HT). My input file was an 1080i m2ts video file (I can provide it to you if you would like).

threads 1, simulate single thread/cpu system, CPU peaked at 100% of 1,600%, encoded approx 6fps:
time ffmpeg -y -i test.mts -deinterlace -r 24 -acodec libfaac -ab 96k -ac 2 -ar 44100 -crf 24 -s 1920x1080 -vcodec libx264 -vpre normal -threads 1 -f mp4 test.mp4
3m20.501s

threads 4, modern quad core, CPU peaked at 400% of 1,600%, encoded approx 20fps:
time ffmpeg -y -i test.mts -deinterlace -r 24 -acodec libfaac -ab 96k -ac 2 -ar 44100 -crf 24 -s 1920x1080 -vcodec libx264 -vpre normal -threads 4 -f mp4 test.mp4
0m54.704s

threads 0, CPU peaked at 800% of 1,600%, encoded approx 30fps:
time ffmpeg -y -i test.mts -deinterlace -r 24 -acodec libfaac -ab 96k -ac 2 -ar 44100 -crf 24 -s 1920x1080 -vcodec libx264 -vpre normal -threads 0 -f mp4 test.mp4
0m35.129s

The Dual Xeon really blasts through the encodes. But even with threads 0 ffmpeg/x.264 is only making use of approx half the available CPU power on the machine, so in this situation with these settings I would suggest running 2-3 concurrent conversions on this machine at once for optimal CPU usage (keeping it maxed out as much as possible while converting as quickly as possible).

If you don't upgrade your server but find ways to speed things up do let us know :-). When the use of GFX cards / CUDA comes to ffmpeg/x.264/?web-M? then you will be able to simply drop a NVIDIA GFX card in your server for 10x faster encodes. There are some cool desktop apps that do this already.

 

The key for a site with a lot of video encoding might be to have a separate server to process the video files.  With a single server, we also have all the other things running such as the webserver and MySQL server so we can not max out the CPU load on our server from just processing video files.

Geeks, making the world a better place
Quote · 10 Jun 2014

I had 162 videos to upload - sizes ranged from 22 mbs to 1.4 gbs. As a whole, it took about  6 hours to get all videos uploaded and processed. Looking at it originally, I did not think this was too bad of a time frame. But looking at it from a user standpoint of someone uploading one or two videos, the conversion time does suck - but it only sucks because everyone is used to the quickness of youtube. Without comparing to youtube, I do not think the time is really that drastic - even for large files. Most with common sense will understand that large files take longer to upload and process, just like they take longer to download. The only thing I would like to suggest would be a circling wait icon or something so that users know that their files are uploading (referring to uploading videos within a module such as Groups).

caredesign.net
Quote · 11 Jun 2014

I was hoping someone would come in and talk about the threads.  There is a old post on Boonex from someone that did increase the threads and saw an improvement.  When I tested it it did not seem to make a difference so perhaps I did not do it correctly.  Dolphin has changed since that old post.

We currently have an eight core processor; If increasing the threads means that our members can see an improvement on processing their videos, then that would be a good thing.

Geeks, making the world a better place
Quote · 11 Jun 2014

 

If you don't upgrade your server but find ways to speed things up do let us know :-).

 This is my testing server above... not my "real" site.

I have a much better commercial server for the site I'm concerned about.

I can't imagine the required server to process lot's of members videos, even if they are not all doing it at once.

IF they were to all do it on a busy day, say the afternoon when my site is going, I'm screwed. 

ManOfTeal.COM a Proud UNA site, six years running strong!
Quote · 11 Jun 2014

 

I was hoping someone would come in and talk about the threads.  There is a old post on Boonex from someone that did increase the threads and saw an improvement.  When I tested it it did not seem to make a difference so perhaps I did not do it correctly.  Dolphin has changed since that old post.

We currently have an eight core processor; If increasing the threads means that our members can see an improvement on processing their videos, then that would be a good thing.

 Im having same issue when too many people upload too many videos and they go in processing at same time it spikes processor and they fail. I uploaded server and got better but still an issue. I will be upgrading again soon but it seems there needs to be a way to limit how many videos are being processed at the same time or it will fail videos. And even then when site gets big enough seems this script is not going to work this way. What have you found out?

Quote · 14 Apr 2022

Hello 

 

One possible way is to limit the threads of Ffmpeg. How to do it:

 

1) open the file flash\modules\video\inc\functions.inc.php

 

2) find the following code like

function prepareCommand($sTemplate, $aOptions)
{
    foreach($aOptions as $sKey => $sValue)
        $sTemplate = str_replace("#" . $sKey . "#", $sValue, $sTemplate);
    return $sTemplate;
}

 

3) change the last line like:

return $sTemplate . ' -threads 1';

 

4) save changes

Quote · 14 Apr 2022

 

 

 I'm having same issue when too many people upload too many videos and they go in processing at same time it spikes processor and they fail. I uploaded server and got better but still an issue. I will be upgrading again soon but it seems there needs to be a way to limit how many videos are being processed at the same time or it will fail videos. And even then when site gets big enough seems this script is not going to work this way. What have you found out?

 I don't check these posts as of late.

ManOfTeal.COM a Proud UNA site, six years running strong!
Quote · 13d ago

Here is a novel idea; don't process videos through ffmpeg if not needed to be processed.

Geeks, making the world a better place
Quote · 11d ago
 
 
Below is the legacy version of the Boonex site, maintained for Dolphin.Pro 7.x support.
The new Dolphin solution is powered by UNA Community Management System.