Dolphin-7.x Resource Hog FLASH xml member_queries browse.php

DosDawg posted 18th of August 2010 in Community Voice. 66 comments.

First of all this has been addressed and basically ignored --> http://www.boonex.com/unity/blog/entry/xml_php_resource_hog_confirmed_

 

it has not gone away, if anything it has gotten worse.

 

i think code satori mentioned there were a few areas where the xml.php file is located on the dolphin platform. the issue i am seeing is coming from basically two locations /flash/xml.php /modules/boonex/xml.php (might be wrong on this one, im tired, been staring at processes run for the past hour. the images will tell the story.

 

I monitored two servers, both are shared hosting environments with ~20 accounts hosted on each one. of those accounts, there are ~7 of those sites that have more than 5 members. there are two accounts that have in excess of 3k members, these are for testing purposes and are not jacked stolen, ripped off with the infamous mods that are sold here for that purpose.

To start with the site(s) images i am depicting have been edited to protect the account owner, i have blottered out the account name for their protection. the first site i am talking about here, has exactly 18 members, and none were online when the RESOURCE SPIKES were taking place.

 

dolphin-7-modules-resource-hog

 

what is the requirement here for the /modules/index.php to hit this type of CPU spike?

here it is again from another account and from same server. except this one shows the /flash/XML.php file acting up as well.

dolphin-7-modules-resource-hog

seriously what could the /modules/index.php need with 71% CPU? note the mysql process was included so you could see which machine this was coming from.

 

dolphin-7-modules-resource-hog

 

dolphin-7-modules-resource-hog

 

dolphin-7-modules-resource-hog

 

 

dolphin-7-modules-resource-hog

 

dolphin-7-modules-resource-hog

 

dolphin-7-modules-resource-hog

 

dolphin-7-modules-resource-hog

as you can see this is from a different server, and just watch how crazy this gets.32% really isnt that bad, and it really is only for a burst. but there is something causing this, and its not site activity, there is something in the application that needs to be looked at. why would a menu query on a site with no activity spike to 32% CPU. this just doesnt make sense. now this site really gets crazy, and i just dont understand this whole ordeal. this has been discussed and those of us who spent our time exploring and investigating were essentially ignored. that could prove to be the case on this go-round as well, but i sure hope not. there are accounts being suspended from the hosts who are listed on the recommend hosting page, irony, accident, or by design. so you grab a hosting package for $4.95/month two months later, you are told  you are suspended for excessive resource usage, and will need to upgrade to VPS. hmmmm, i will not elaborate on this because that is not what this is about to me at this moment, but you have all read the posts in the forums where members were suspended and told they would need to upgrade to VPS.

 

dolphin-7-modules-resource-hog

/flash/XML.php --> /modules/index.php --> /member_menu_queries.php. resource usage not all that bad  you might say on this one. not really, but again, there is nobody on this site, there is no activity on the site. so what is causing this?

dolphin-7-modules-resource-hog

 

without running an actual timer on these occurrences, i would guess that the time between spikes is about 1 minute. same files spiking the CPU.

 

dolphin-7-modules-resource-hog

 

dolphin-7-modules-resource-hog

 

now this spike is not something i see often, but thought it interesting the CPU spike with no users on this account at the time of the spike. logically how many people would you think would need to hit this blog at one time, in order to  knock off this type of resource? i dont have any idea, but these are not issues that can be managed from the server-side. this is directly related to the source code.

 

dolphin-7-modules-resource-hog

 

and i dont have a clue what the heck happened here. browse.php seriously guys, what would cause this, it certainly isnt traffic.

dolphin-7-modules-resource-hog

 

this is just an idea of what resources are used on the cron jobs. and if setup based on the required suggestion from the developers, to run every minute on the minute, this is heavily resource intensive. what is causing the CPU usage on these cron jobs? since they are running from the database, shouldnt they be more optimized than this? how come this only started with dolphin 7, this was not an issue with Dolphin 6?

View the Images offsite

 

this is not a bug report, this is not a complaint, this is a blog about what i see going on with the Dolphin 7 platform. there was an instance where we were told to turn off the profiler, then told to turn on the profiler. which is it, and what is causing these CPU spikes?

 

hopefully we can get some resolution. AlexT or any other developer who wishes to have access to my servers, please inquire, you can certainly take a look. i would also like to see the processes  that are running on the demo-dolphin site you guys run, because there were times when it was pegged out i am sure, and you have a great deal of members on that site. but a site with 18 members, should not be spiking the CPU +20% muchless the astonomical amount that you see depicted in these images.

 

Regards,

DosDawg

HoustonLively, i had to edit my post in order to provide a definitive answer.  since you are not allowed to add images to the post comments.

Disabled All Versions of IM

HL as soon as i read your inquiry, i went to check on the status. The above is what was being used. Then i just sat there and watched and it took ~15 minutes, but the results were astonishing:

 

Resource Hog Dolphin 7 appears to be IM modules

 

so as you can see, even though you killed all instances of the IM, there is still something running the /flash/XML.php file, and its still spiking the CPU. Weird thing is this is ~76 lines of code, and using 70% of the CPU.

 

I have this strong suspicion that the ilnes 17 - 22 have something peculiar to do with this CPU spike. Since the security mechanism was broken upon release, and nothing has been done to fix it, we were only told to disable it, and that they would get to it at a later date.

 

define('BX_SECURITY_EXCEPTIONS', true);
$aBxSecurityExceptions = array(
'POST.data',
'REQUEST.data',
);

 

now to see the entire /flash/XML.php file

<?
/***************************************************************************
*
* IMPORTANT: This is a commercial product made by BoonEx Ltd. and cannot be modified for other than personal usage.
* This product cannot be redistributed for free or a fee without written permission from BoonEx Ltd.
* This notice may not be removed from the source code.
*
***************************************************************************/
$sModule = isset($_REQUEST['module']) ? $_REQUEST['module'] : "";
$sAction = isset($_REQUEST['action']) ? $_REQUEST['action'] : "";

if(($sModule == "mp3" && $sAction == "screenshot")
|| ($sModule == "board" && $sAction == "transmit")
|| ($sModule == "games" && $sAction == "makeShot")
|| ($sModule == "livecam" && $sAction == "channelShot")
|| ($sModule == "photo" && $sAction == "transmit"))
{
define('BX_SECURITY_EXCEPTIONS', true);
$aBxSecurityExceptions = array(
'POST.data',
'REQUEST.data',
);
}

$sGlobalHeader = "modules/global/inc/header.inc.php";
if(!file_exists($sGlobalHeader)){ header("Location:install/index.php"); exit;}

require_once('../inc/header.inc.php');
require_once($sGlobalHeader);
require_once($sIncPath . "constants.inc.php");
require_once($sIncPath . "db.inc.php");
require_once($sIncPath . "xml.inc.php");
require_once($sIncPath . "functions.inc.php");
require_once($sIncPath . "apiFunctions.inc.php");
require_once($sIncPath . "customFunctions.inc.php");

$sModule = empty($sModule) || !secureCheckWidgetName($sModule) ? GLOBAL_MODULE : $sModule;

$sContents = "";
$sContentsType = CONTENTS_TYPE_XML;

if($sModule == GLOBAL_MODULE)
{
require_once($sIncPath . "xmlTemplates.inc.php");
require_once($sIncPath . "actions.inc.php");
}
else
{
$sModuleIncPath = $sModulesPath . $sModule . "/inc/";
require_once($sModuleIncPath . "header.inc.php");
require_once($sModuleIncPath . "constants.inc.php");
require_once($sModuleIncPath . "xmlTemplates.inc.php");
require_once($sModuleIncPath . "customFunctions.inc.php");
require_once($sModuleIncPath . "functions.inc.php");
require_once($sModuleIncPath . "actions.inc.php");
}

switch($sContentsType)
{
case CONTENTS_TYPE_XML:
//--- Print Results in XML Format ---//
header("Cache-Control: no-store, no-cache, must-revalidate");
header("Cache-Control: post-check=0, pre-check=0", false);
header("Pragma: no-cache");
header ('Content-Type: application/xml');
$sContents = "<?xml version='1.0' encoding='UTF-8'?>" . makeGroup($sContents);
break;
case CONTENTS_TYPE_SWF:
header("Content-Type: application/x-shockwave-flash");
break;
default:
break;
}
echo $sContents;
$oDb->disconnect();
?>

 

then looking at this file, /flash/XML.php i see where it calls /modules/global/inc/header.inc.php and this file has what looks like a bunch of garbage code in it. why would you need to define the database parameters again for this file, and what is this exactly doing, and why. there is no user parameters, and there are no actual database queries here, so why would you need to have all of the database parameters here again, when  you have them in the /inc/header.inc.php file? why would we just not require /inc/header.inc.php in this instance.

 

/**
* Data Base Settings
*/
if(!defined("DB_HOST")) define("DB_HOST", $db['host']);
if(!defined("DB_PORT")) define("DB_PORT", $db['port']);
if(!defined("DB_SOCKET")) define("DB_SOCKET", $db['sock']);
if(!defined("DB_NAME")) define("DB_NAME", $db['db']);
if(!defined("DB_USER")) define("DB_USER", $db['user']);
if(!defined("DB_PASSWORD")) define("DB_PASSWORD", $db['passwd']);
if(!defined("DB_PREFIX")) define("DB_PREFIX", "Ray");
if(!defined("GLOBAL_MODULE")) define("GLOBAL_MODULE", "global");
if(!defined("MODULE_DB_PREFIX")) define("MODULE_DB_PREFIX", DB_PREFIX . ucfirst($sModule));

 

does that appear to be a bunch of garbage that is essentially useless?

 

there's some more of the story. i do see that the ones who should be reading this are avoiding this post. interesting to say the least. i think i will go invite the powers to come take a looksee.

 

 

 

 

 

 

 
Comments
·Oldest
·Top
Please login to post a comment.
dolphin_jay
Again very nice write up from the last time you and i tested this almost 5 months ago. I just told one person that got suspended to un install simple messenger and there load went from 27% to 7% i think it was and they only had 15 members or something like that.

I then told them to un install the regular messenger to see what the load would be. have not heard back yet.

You know the new server i built at home had the same problem and i un installed those two modules and now i run at see more like .97% Yes point 97% on a quad core amd running at 4ghz and 8 gig or 1600mhz ram.

So if two little silly modules can bog down a computer of this size then AGAIN i have to say there is something wrong. I wish i could just tell you how and where to fix it but, it truly needs to be fixed.

I also had the same thought in mind - was this built in, was it by mistake, was it just a bad string of code some place... who knows.

If I ever remember to take some screen shots of my cpu load on a server thats not even online by the way I will post them as well...

Thanks again for the nice pics and info.
houstonlively
Do you suppose this is related to the tons of cron emails I get about Mysql errors. It's not like my site has any traffic.
CodeSatori
If it seems like too much to tackle, outsourcing the fix may be an option worth looking into. I'm sure there are developers who can crack it straight for a reasonable price, and considering the trouble it's causing to customers here, the investment would no doubt be worth every penny.
dolphin_jay
@CodeSatori

If i had the funds to fix this "BAIT and SWITCH" tactic, i would do it for sure. Any idea on what you would charge for this ?
houstonlively
My access log gets filled by entries like this, which occur every 5 seconds.

xx.xx.xx.xx - - [18/Aug/2010:13:31:00 -0400] "GET /flash/XML.php?module=im&action=updateInvite&recipient=3&_t=1282152666655 HTTP/1.1" 200 70 "http://mysite.com/" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8"

I disabled the IM module to see if things get any better
CodeSatori
"updateInvite" in /flash/modules/im/inc/actions.php queries the database, here's the relevant code. Comments to follow.

$rResult = getResult("SELECT `SenderID`, `Message` FROM `" . MODULE_DB_PREFIX ."Pendings` WHERE `RecipientID`='" . $sRspId . "'" . $sBlocked . " ORDER BY `ID` DESC");
//--- if there is a message return it and some information about it's author ---//
while($aMsg = mysql_fetch_assoc($rResult))
{
$aUserInfo = getUserInfo($aMsg['SenderID']);
$sContents see more .= parseXml($aXmlTemplates['message'], $aMsg['SenderID'], $aUserInfo['nick'], $aUserInfo['profile'], $aMsg['Message']);
}
CodeSatori
In the above and the surrounding code inside the same "updateInvite" action, if you follow the trails, here's what happens when it runs:

- Database query for new messages
- Database select query inside function getBlockedUsers()
- Database select query and data processed inside function getUserInfo()
- Data parsed into XML via function parseXml()
- Database delete query

If you do all of the above at 5 second intervals, and dozens of users are online, it will definitely take its see more toll. Remedies off the top of my head:

- blockedUsers and userInfo data should be cached, not requested again and again
- delete query can be removed if message select query narrows by > datetime since last query, cleanup left to a cron job

This would drop SQL queries from 4 to 1, which is a dramatic difference in terms of performance. Moreover, actions recurring on rapid intervals should be isolated into smaller interdependent file subsets, and called as such only, to escape the otherwise large overhead.
CodeSatori
@dolphin_jay: Here's me wondering if I just cracked a good part of it in the last 15 minutes. With reference to my note on "smaller file subsets" that handle this independently, XML.php is a good illustration of the problem:

If you open up /flash/XML.php and read through just the files it requires — no less than 16 external files required, and I dread the thought of going to look them all up to see how much processing overhead they may produce, aside the obvious memory consumption see more from all the unused functions loaded into memory.

Call the above action (four SQL queries) through the heavy XML.php at 5 second intervals, multiply by a few dozen users, and it's a small miracle if your server isn't crawling.
houstonlively
This happens every 5 seconds even with the IM module disabled. (I'm using URL Snooper to watch it) I don't get it. It's a pretty good bet that there will be no new messages with the messenger module disabled.
dolphin_jay
Very Nice will give it a go in a few. Just playing with a wiccle first !
CodeSatori
The 'updateInvite' action calling XML.php is called from /modules/boonex/messenger/classes/BxMsgModule.php at line #138 in function serviceGetInvitation(). See what happens if you comment out the javascript making the call.

Strange thing is, I can't find any other references to 'updateInvite' or 'serviceGetInvitation' in the code, including no explicit call to serviceGetInvitation() for creating the javascript that should call the 'updateInvite' action.
houstonlively
I must confess complete ignorance as far as IM and messenger stuff goes. It's something I have given zero thought until today.

So... I have a Boonex 'Messenger' module, a Boonex 'Simple Messenger' module, and a Flash IM module.

Can someone explain to me wtf we need all these messenger modules for? Uninstalling the 'Messenger' and 'Simple Messenger', and disabling the flash IM seems to have shut everything up. I'm quite content to leave it that way.

Dawg, can you check to see if my see more resource usage has dropped any?
DosDawg
HL,
i posted a response to your inquiry, back to the original post, because i am not allowed to provide images here. so go back to the original post, and you will see two screen shots. not as bad as it was yesterday, but not what i would consider great either. the /flash/XML.php file is spiking ~15 minutes. so there is still something going on.

i also added some information on what i was seeing in the /flash/XML.php and what it requires, then looking at some of the files that are required, the see more unnecessary defines that are on the /modules/global/inc/header.inc.php file.

at any rate. sure wish we can attract boonex attention to this.

Regards,
DosDawg
CodeSatori
@houstonlively: I suspect all the different messengers are communicating between themselves, because you're not talking to them! Perhaps something new is emerging in the darkest abysses of the codebase... It may be AIRH, or an Artificially Intelligent Resource Hog. I'd get one of those anytime, put it in a cage, and watch it hog around all day long!
LeatherSportB
I think there is a TON of room for improvement for what amounts to AJAX requests for notifications and chat messages.

CodeSatori is on the right track with what he found. Something to think about, XML parsing can use a lot or resources. For the type of data that is being returned are all of these includes, functions, templates, and parsers really needed?

What if there was a very simple 'pending notices' table and instead of issuing multiple requests for different information across may tables see more requiring lots of steps every few seconds, the client simply asked 'is there anything waiting for me?'. The notice table could even contain a 'pay load' field so there is no need to do an additional request if there is something waiting.

I see the issues as two fold, updating the server side to this new table and updating the client so you only have one thing polling the server (kind of like Apples push notifications) that can check for notifications, then trigger the required client side feature.
houstonlively
@ CodeSatori You are probably right about the messengers talking to each other. I should probably also delete their files from the server so they don't attempt to reinstall themselves. I any case, if I ever decide to launch a DOS attack against myself, I now know to just reinstall the messenger modules.
DosDawg
I have invited the following boonex Staff to review this with us. hopefully they can find a moment to help us out:

Andrew
AlexT
Igor
AntonL

of these four, we would hope that we could find an answer with this issue, and not declare it as an enhancement to not have the CPU spiked on the server.

Regards,
DosDawg
DosDawg
wanted to make sure this discussion is picked up in conjunction with my ramblings. there are many who as deano states, know about this problem, and there has not been a fix for it. when this was mentioned earlier to one of the boonex team members, i was told they were working on 7.0.3 bugs, and getting that ready.

well the problem here and i was actually told thank you, is that this has been a persistent problem since the release of Dolphin v7.0B, it has been widely discussed and swept under see more the rug so-to-speak, so something is weird here.

we are awaiting the answer on this, and there are many who will be very happy when resolution is found for this problem.

REVIEW please --> http://www.boonex.com/unity/forums/#topic/load-on-the-server.htm

Regards,
DosDawg
God grant us all patience
sorry for this post
houstonlively
You should be. It's pointless.
AlexT
PHP consumes as much resources as possible to execute script faster. So it can take 100% of one CPU, but it takes only 0.1-0.5 seconds.
So - any php script consumes some considerable amount of CPU when it is executing, but it is lasting very fast.

The reason that you see XML.php in the processes is that this file is executing more often than others (every N seconds from each user) ... since it is executing more ofter it is more likely that this file will be in processes list.

I will try see more to have a deep look at this call, to see how to optimize it, but it can not be done too much. The most useful things in this case are increase delay between calls (from 2 sec to 5 sec for example) and optimizing the server by installing some php accelerator and make sure that mysql cache is enabled.
DosDawg
AlexT,
thanks for the effort of the response. I am not by any means trying to start trouble here, but your answer is really not acceptable. first of all, it appears that you have not even read what was posted. the sites this is happening on, are sites where there is no traffic, 18 members on the one site. so what would happen if this site were to grow to 5k or 10k members, what would happen to the CPU if the XML.php is going to consume 100% for any given period of time, and its longer than 0.1- see more 0.5 seconds.

ok so every N seconds for every user, this is not the truth either or from my perspective its not what is going on here. there are other dolphin sites that never spike the CPU. and your mention to optimize, and suggestion to use some php accelerator, the requirement of a php accelerator was not mentioned in the application requirements. in actuality, i think i am running eaccelerator on both of these machines, but will check and make sure.

..... 2k character limit got me
DosDawg
i think the optimization is not required on the PHP as much as its the XML that is being output to. i guess the other component here, is why did this start with Dolphin 7, this was not a problem on D6.1.6, so what exactly is the XML.php (multiple of them) actually for, and why could you not trim down the calls, most specifically on a site with only 18 members.

I am not clear on your explanation that PHP consumes as much resources as possible. I dont think this is factual, i can test this, and see more see, but what would make any other PHP script run and not spike the CPU resources? There is something wrong with this code, its not PHP, its not the lack of a php accelerator, its the programming that was done. there is a clear and evident problem.

then you go on to mention mysql cache. exactly what is that going to do for this specific resource usage problem? and where you have attempted to walk around the root of this problem, you mention the /flash/XML.php but what about the other FILE(S)
/modules/index.php
/member_menu_queries.php

why are these files also hogging the resources?
dolphin_jay
I think CodeSatori is spot on here. There is so many files being called into memory and using cpu power for no good reason at all. If i un install a module or disable a module whats the point of having it even fire the code off in the first place.

There needs to be one simple check first. Is the module installed and being used if not then move along to something else or die...

HL had a good point as well do we really need 5 different ways to chat on 1 site. Sure everyone wants to be see more like face book with the cool little box down there.

I say install simple messenger and IM on unity so we can talk on here live with people that are on our friends list and see what happens to your servers then !

Its only fear if we have to put up with this poor coding in this section of dolphin you should have to as well.
cbassthefish
@dolphin_jay: I did ask whether they would be switching on the simple messenger on one of Andrew Boon's blogs for this reason. Alas I had no reply.

http://www.boonex.com/unity/blog/entry/Ongoing_Update
sandalsonly
That is a good point dolphin _jay! What would happen if we had messenger here? And we don't need all the ways to chat for sure, but it should be a choice if we wont to or which ones to use.
The chat room sucks on my site, because no one knows who's in there or when they are in there, so everyone uses the shout box, and then if its personal, they use the simple messenger. I removed the video messenger and I may remove the damn chat room because its useless!
I keep removing modules left and right see more also, just to reduce the power, for long its just going to have forums and blogs, and member pics. But I don't think my members will pay for that.

This is a good topic,
thanks, Dos Dawg,

Derrick
CodeSatori
@AlexT: All scripts take as much of CPU as they need to run; whole the point is to make them light enough to not cause an issue. If it takes up to 0.5 seconds per request, it's obviously far too heavy for a recurring context like this, and won't scale up in a multi-user environment. 10 users means the query is made 2x per second, and that alone takes up the full CPU time. What if you have 50 users?

If you read through the files associated with action, it's obvious that there's a huge amount of see more optimization that can be done. For example:

- Add in an internal session cache to remember the results of the getBlockedUsers() and getUserInfo() queries -- two SQL queries off.
- Remove delete query, get new messages narrowed by time last queried, and have a cron job do the cleanup at set intervals.
- Go over the required files in XML.php, collect the necessary functions into a subset of files, and then make the calls through your "light" version -- without huge unnecessary overhead.
houstonlively
Every logged in user polling the server every 5 seconds, is just crazy.... especially when most of those requests are going to return empty results. Instant messaging needs to be implemented differently. Boonex needs to look at a long polling solution like Meteor.

http://meteorserver.org/

It will require a unique server setup, but just glancing through the docs, server setup looks quite simple. Obviously, not all hosting companies would be able or willing to do this, so a standard short see more polling IM should remain an option.
LeatherSportB
What hustonlively found, meteorserver.org, is whats needed for chats and notifications to have a chance to scale. Meteor is A LOT like push notifications, but for web browsers, not iDevices.
dolphin_jay
the simple im the regular im and the spy features need to be rewriten from scratch to be much more lite weight then they are.
pierrehs
I my site was hosting at Dolphin 7 Arvixe with Personal Class but they suspended my account for excessive CPU resource
See photo
http://ups.imagup.com/ano1/1282399623.png

Now I am with Business Class and four times more expensive and if BoonEx can not solve this big problem I migrate my site of Dolphin on script less potent as vldPersonals or Social Engine
but I hope a solution will be found, because I like Dolphin
CodeSatori
@dolphin_jay: Assuming the cross-function dependencies aren't totally over the top, it should also be possible to just remodel the existing code into a tight pack without the overhead. It seems to me that the bulk of the performance hit is brought about by what's essentially clutter in the context.

However, there's little point in the community preparing ready fixes for issues like this, if the software developers themselves can't see the problem. Give a man a fish, and all that -- would be sound see more if considerations like this were actually an integral part of their development approach.
dolphin_jay
Yes i agree with you completely here. Would be nice to just rip that out as i do not use those function.

I guess it really is up to the coders of dolphin to either provide a fix or just let it go like its been since december....
dolphin_jay
Well maybe monday will will have a better answer here.....
DosDawg
server login has been provided to AlexT, and hopefully he can provide some insight. he seems to think its my server configuration. alas! what about those servers elsewhere that have this same problem, is that a problem with my server configuration there as well?

Regards,
DosDawg
CodeSatori
The problem with your server configuration in this case is probably that it's not running 16 cores and 64 GB of RAM. However, that's not exactly listed in the requirements for this software.
houstonlively
Dawg, for what it's worth, uninstalling all the messenger modules on my site seems to have stopped all those cron error emails I was getting. I didn't need to send IMs to myself anyway.
AlexT
@DosDawg

Nothing was provided to me yet!
DosDawg
@AlexT
the email was sent Aug. 20, 2010 4:40 AM EST, to the boonex.com email address you provided me in the PM. can you check in your garbage can, or provide me with an alternative email to forward this one to?

Regards,
DosDawg
DosDawg
im not really even sure that the email address you sent me is actually working, i request a delivered receipt for the email i sent, and that has not been sent to me, so that could almost mean that you are not getting the mail? please let me know if you are not getting the emails, i have sent it twice now for you. please provide an alternative inbox other than boonex.com because for whatever reason, my mail is not making it to that box.

Regards,
DosDawg
AlexT
@DosDawg

It was in spam folder for some reason, only now I've found it.
But unfortunately the access you provided is not working.
DosDawg
responded in the email. sorry about that one. you should be able to get in now.

Regards,
DosDawg
Nathan Paton
I'm seeing that this is becoming more of a challenge. Will our heroes conquer the perils of email filtering algorithms? Find out in the next episode!
CodeSatori
Recommended reading: "The Quest for the Broken XML.php" by B.O.O. Nexien.
AlexT
@DosDawg

I've got the access now, investigation is in progress...
Nathan Paton
Godspeed, AlexT. Godspeed.
houstonlively
Alex, if you ever need to investigate the Mysql errors, I can reinstall my messenger modules and supply an abundance of such errors for your perusal. Timing is critical though.... at least one user must be logged in to produce these errors.
DosDawg
@AlexT,
many thanks for your time. working on suggestions, and will keep this post updated if there is any significant difference in performance with utilizing the cache system on mysql. however, the mysql process is not the one that is ripping the CPU, but willing to try anything.

your time is valuable, and your assistance in this matter is greatly appreciated from the community.

Regards,
DosDawg

@CodeSatori,
you are invited as well to investigate my servers, I've installed Munin at see more the request of AlexT, but quite frankly, i have no idea what these graphs actually indicate. LOL
CodeSatori
@DosDawg: Thanks, I got the messages — it's mid-week and I'm pretty swamped with work right now, I'll skim through them as soon as I get a chance and bounce some thoughts back. Overrall I stand by my take that this is primarily a software issue, not a server configuration issue.

You may be able to fine-tune your server so as to keep the process from going totally out of control, but this doesn't really address the actual issue, which is in the bloated process inside Dolphin itself causing this see more to begin with. Would be grand if someone officially recognized it (because really -- just look over the code, it's a common sense observation that it's bloated), and tried to fix the root issue itself.

In Finnish, "Munin" means "I am laying eggs", and also "I am screwing up". That's the depth of my exposure to Munin, really, so no comments there.
AlexT
Investigation showed that there are some inefficient server configuration settings that can cause high server load.

1) suPHP - this main part of the problem, this is most inefficient PHP configuration also known as CGI (suPHP is the same CGI with exception that every request is running under particular user ID, it adds additional level of security). With a lot of small requests it causes high load on the server, because apache has to load php entrepreneur and run separate php process each time see more small request is made. CGI is about 5x times slower than mod_php(dso) for small requests. Also opcode cache compilers do not work in such server configuration - so there is no way to even optimize it.

I understand that shared hosting has to be more secured and mod_php maybe is a bad choice, but there are some more efficient and secured configurations available, for example MPM_Prefork+SuPHP+FastCGI+eAccelerator.

2) MySQL is not optimised, enabling query cache can speedup mysql about 5x time faster, other servers where this setting enabled showed that %80 of select queries are taken from cache. Also there are many other mysql settings which can speedup MySQL server.

3) Backups are inefficient. Backup process is archiving whole system each day, it causes that up to 50% of the server resources are spending on backup during several hours. During this time sites can experience considerable slow down. I suggest to implement incremental backup solutions.

Dolphin has ways to optimize it too. We are going to implement optimisation on the whole script in 7.0.3 and in future versions.
DosDawg
AlexT,
the time you spent looking at my two servers was greatly appreciated. However, you were looking in the wrong place, because I have no other HIGH CPU usages on anything except the dolphin script. So pointing out that there are inefficient configuration settings, is 100% inaccurate. You mentioned to me that there were apache modules that can cause high load, keyword there 'can', but they are not, your statement above says 'inefficient server configuration settings that can cause high server see more load' again, keyword 'can', but they are not. The absolute of this is that Dolphin is causing the high CPU resource usage, not once are you addressing this.

Due to the fact that Dolphin is primarily insecure, and where there was a feable attempt at securing the application, that blew up in our faces and we were told to disable the security feature, that it wasnt working, and that it would be addressed at some later date. The likelihood of Dolphin becoming vulnerable to compromise is extremely high. suPHP is but one method of securing the server to protect all accounts on the server.

Item 2) you state that MYSQL was not optimized, that was on one of the servers, so that statement is incorrect and extremely inaccurate, because on the one server where MYSQL is optmized, there is the very same problem with the CPU resource usage on the XML.php file as well as the members_queries.php or whatever that file is, then there is the other /modules/boonex/index.php. Again, you address everything here but the problem, and the problem are those files.

.... 2k char limit
DosDawg
Then you go on to accuse my backups of being inefficient, LOL, my backups have nothing to do with Dolphin HOGGING RESOURCES sir. The backups are not a problem, there is no speed problem with the server, and the backups are ran in off-peak hours, so your insinuation that there is a problem with my backups, that is causing Dolphin files to HOG the CPU resources, is ludicrous. Again AlexT, you failed to address the problem here, you are pretty good at pointing the finger at other issues, or potential see more issues, but you are not addressing the problem. THE DOLPHIN FILES HOGGING RESOURCES.

FYI, there has been no change in the situation, Dolphin is still a resource HOG, and there is nor has there been a reason provided. because a file calls the database N times a second, that is what needs to be fixed. I asked where this setting could be changed, I received no answer to this.

Regards,
DosDawg
AlexT
I have pointed you the problem ! If you do not want to follow the recommendations - your choice !
I have explained to you what is the problem many times.

We can and we will optimize dolphin performance, but all we can do is about 10%-50% of performance increase, and you as hosting provider can do more than 1000% by changing server configuration.
CodeSatori
@AlexT: If you take a good hard look at the code surrounding the XML.php requests, you should be able to get more than a 500% performance increase by writing it over from a scratch, and more than 250% even if you just slimmed it down and removed all that processing overhead. (I think we have all been expecting for notes on what you intend to do about optimizing the script, if anything.)

Just to be clear here: The problem here is with an inefficient script, and while many (common) server configurations see more can escalate the problem, the server configurations themselves are not the problem. They merely highlight a problem in the software design. If you require a particular fine-tuned server configuration for certain features to work, then it should be clearly documented in Dolphin requirements.
Nathan Paton
I suppose your recommended hosting providers don't actually support Dolphin since this issue is clearly one of server configuration.
houstonlively
Alex, perhaps you should just add 'Dedicated server required' to the list of requirements to run D7, because trying to manage a site on a shared server that ISN"T running suphp, is a real pain in the ass.... especially if you don't have shell access. You always end up with a bunch of files written by apache user 99 that are impossible to delete, which makes a person want to pull their hair out. Suphp is about more than server security. It makes things a lot easier for the webmasters also.

The see more simple fact here, is that the IM modules are resource pigs. Fix them.
CodeSatori
@houstonlively: I released a GPL browser-based file manager script a while back that you can use to easily get rid of Apache-written files. It's a common problem alright, and one I've often cursed over myself. Download here: http://www.wiccle.com/page/download_fileman (4KB / plug and play / one file only)
DosDawg
@AlexT,
no sir, i have in fact made sure that your suggestions were heeded. however, you persistence on the fact that every other application, module, or otherwise, anything remotely related to dolphin would be the cause of the excessive CPU resources on specific Dolphin files, is what i find to be a tad annoying. Yes you pointed out some issues you considered would help with optimization of the server, however, I can show you on dedicated servers where this problem is in existence as well.

yes see more the shared environment runs suPHP and that is the recommended configuration when running a shared or semi shared environment. changing to DSO would be particularly insane and try to run a shared environment. Your ideology is that the server should be stripped down, and that only minimal dolphin required and specific apache modules be loaded, and that this would regress the high CPU usage that the dolphin script is demanding?

Nowhere sir, is it ever specified that php should have a specific configuration, and regardless of the configuration, I think this high CPU usage will persist, but i will take your advice, and will show you that no matter what is done to a server, stripped down, nothing running but dolphin, matter of fact, i think deano has said this already, there is an extremely high CPU usage, but i will do as you suggest, and will setup an environment with minimal modules for apache, and minimal modules for php, optimize mysql, where i dont see the real gain in that, because there are a ton of queries that are reported as too large to cache, so you only get about 1/8 of the queries to cache at all.

i do believe that some of these tables can be optimized, and that there should not be queries on the SELECT * on any of the tables, you should optimize your queries, so that the database is being called for specific information that will not require extra stress and load on the CPU or the mysql server.

Regards,
DosDawg
DosDawg
@Magnussoft
that would be a very good presumption. since we see every day in the forums where a member has been suspended for excessive resource usage, and most times they have less than 20 members on their site. but as elaborated by AlexT, this is a server configuration issue, however, the exact server configuration that is required by dolphin 7 as it stands today, has not been revealed. hmmmph

Regards,
DosDawg
houstonlively
@ CodeSatori Thanks for the download. I have a vBulletin site on a shared server that gives me constant headaches due to this issue. I had been using a php shell script to delete the files, but the hosting company recently installed some annoying security product, and the script now sets off alarm bells if I try to use it.
houstonlively
Unfortunately, the script doesn't work on the site I need it to work on. Like I said, they installed some security product that stops scripts like this from working.
CodeSatori
Custom server security setups are a dark cloud hovering over the heads of application developers. Can you message me info on what exactly doesn't work, so I can see if there's an alternative way to do it that might work?
how many adds per cent productivity such a bunch MPM_Prefork+SuPHP+FastCGI+eAccelerator.?
who uses it, there are examples?
wickiewac
I'm eager to hear a resolution to this as well. My host just contacted me about my site having very high resource usage and if this is expected then I need to purchase a different package. My site hasn't even Launched yet and I only have 3 users helping to test it. This high resource issue is very concerning and disappointing.
My host is one of the Top Providers listed and they suggested shutting down modules and loading 1 at a time to find the culprit. I guess I'll start with the Messengers.
I'll see more check this thread again periodically for updates.
..

wickie
jimmybo
I've been running my site Dolphin 7 site with no problems for over a year (upgrading to newest version always). Today for the first time I got notice from my host that they had locked down my /flash/XML.php file due to "extreme" server resource use. It seems this is still a problem even in D7.03. :-(
sexycatzeyez
Funny, 7.07 is out now and still the same issue.My host was ok until I installed 7.07 a month ago. I just got a warning from my host and of course the offending file was flash/xml.php. I'm already looking for a better script if this thing couldn't be resolved from the time this thread started , through 5 updates, til now.
 
 
Below is the legacy version of the Boonex site, maintained for Dolphin.Pro 7.x support.
The new Dolphin solution is powered by UNA Community Management System.
PET:0.08704400062561