First of all, I want to thank DreamHost for announcing memCron as one of the two grand prize winner for its first ever API contest (congrats to Dreambot for tying the grand prize!). This is the first time I actually won something for a program I wrote, so thank you for recognizing the quality of the script I developed! 
The announcement of the API contest results finally brought in many new users who have never heard about memCron up till the announcement. Luckily memCron site survived the spike of visitors without any major issues. I was actually wondering if the announcement will put up a stress test for the memCron script, but in the end the memory usage didn’t change that much at all (probably due to WP Super Cache). Instead, all I saw was a spike in CPU load, jumping the load from 0.2 straight to 13 right after the news was posted on the DreamHost blog. You can still see the spike in this graph generated by memCron. 
Many of these new visitors were brave enough to give memCron a try and the feedback so far have been mostly positive. Some of the feature requests are pretty insightful and I managed to spent some time to code them into memCron 0.4. After working on it for the past few days, I think it is ready to be released. Please give memCron 0.4 a try and let me know if it works just the way you imagined it would be.
Here is a list of notable changes since version 0.3 (or check out the detailed changelog):
- Added: New parameter
$max_memoryand$min_memoryto limit the memory range that memCron should operate within - Added: New parameter
$mem_thresholdto set a memory threshold, which when exceeded, will send out notification to$email - Added: Base memory and swap memory size to function output
- Changed: Tooltip for the resources usage graph is now cuter i.e. in yellow!
- Removed:
$debugand$usernameparameter from config.cfg, now DreamHost API only needs API key for authentication - Fixed: Memory target are now computed more correctly during downsize
- Fixed: Total memory is now displayed correctly on the resources graph by accounting for max swap memory, which is capped at 450MB
Download: memCron 0.4 (31.95 KB)
If you find any bug in memCron 0.4, please post them here and I’ll get them verified and fixed asap. If you love how memCron saves you time and money through its innovative memory management system, please consider donating to support this project, thank you!
As always, I’m open to new feature request so don’t hesitate to send them in. I’ll try my best to add the most requested feature into the next release! Let’s make memCron the best damn DreamHost PS app in the world! I know it can be done with your support! 
{ 113 comments… read them below or add one }
Thanks, terrific!
Awesomeness
Thank you very much! But still no timezone support?
Hi kwurzel, right now you can go into
index.phpand change the$gmt_offsetto the one that correspond to your timezone, for example the current value of -4 corresponds to EST with daylight saving.I will move the
$gmt_offsetto the config file in the next version. Thanks for your feedback.I am confused as to why there is a setting in the config.cfg file for “cron_interval”, isn’t this being set via the Dreamhost Panel (Goodies>Cron Jobs) anyways?
Also I notice that Dreamhost’s panel wouldn’t let me set anything lower then every 5 minutes, if I tried to set it to every minute or 2, or 3, the panel would say “Too many minutes selected!” and not allow the Cron Job to be added.
I haven’t tested this yet but I wounder if it would be possible to set the cron job via SSH using “crontab -e”, and possibly allow you to set it to run every minute?… Has anyone tried this?
Thanks for the Great script, and Congrats on the Contest Win
Hi BassKozz, the
$cron_intervalvariable is used in the memory scaling algorithm i.e. changing this has an effect on the calculation results. Setting it in DreamHost panel only tells DreamHost how often to run memCron, but memCron itself needs to know how often it is being run to get things right.You are right that DreamHost doesn’t allow cronjob that is less than 5 minutes apart, that is part of the reason why memCron’s default interval is 5 minutes. I think DreamHost discourages running a cron job too often as it may cause stability issues to your server.
If you do choose to be aggressive, then nobody is stopping you from running memCron every minute using crontab, I haven’t tried it myself but it should be doable. Do take note that memCron takes more than 1 minute to run each time because it will attempt to collect 60 memory samples over 1 minute. This could lead to file locking / corrupt issue when 2 memCron instances overlap.
Thank you very much for you help. That’s totally sufficient for the moment! But anyway, i’m looking forward to 0.5.
Looks good with all the features I wanted.
While there might be things that need to be added, I would suggest being conservative since we don’t want memcron to be using lots of resources on the server.
There is a bug with the graph: When I open index.php in Internet Explorer 8, the actual graph lines don’t show, though the axes do. Everything’s ok in Firefox.
Hi Zack, I understand your concern about bloatware, personally I wouldn’t enjoy maintaining a project that is overcomplicated than it needs to be as well.
Just to reassure you, new features will only be added if many users requested them and only if they make memCron easier and safer to use. 
As for IE8, right now you will need to click on the compatibility view icon beside the address bar to see the graph. This is a problem related to the excanvas and flot JavaScript used to plot the graph. Once a newer stable version of flot is released I’ll update memCron to support that.
Hello!!
Great application, I was ready to make a nice donation, when I started receiving error emails from Dreamhost:
"Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /home/xaxa100/portofoli.com/memcron.pl line 40.BEGIN failed--compilation aborted at /home/xaxa100/portofoli.com/memcron.pl line 40 (#1)
(F) You said to do (or require, or use) a file that couldn't be
found. Perl looks for the file in all the locations mentioned in @INC,
unless the file name included the full path to the file. Perhaps you
need to set the PERL5LIB or PERL5OPT environment variable to say where
the extra library is, or maybe the script needs to add the library name
to @INC. Or maybe you just misspelled the name of the file. See
perlfunc/require and lib.
Uncaught exception from user code:Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /home/xaxa100/portofoli.com/memcron.pl line 40.
BEGIN failed--compilation aborted at /home/xaxa100/portofoli.com/memcron.pl line 40.
at /home/xaxa100/portofoli.com/memcron.pl line 40"
Any clue? It would be grateful if you could give a hand..
Hi padelis, the error message you received is related to perl module not found, which shouldn’t happen. Try to see if the following file exist on your server, if not you should ask DreamHost tech support about it:
/usr/share/perl5/Statistics/Descriptive.pmI am having the same trouble but all I received from DH support is something about using CPAN to manually install whatever this is. Is this problem common? Haven’t been able to figure out a fix yet.
DH staff installed the statistics module but I am still having some troubles without Crypt::SSLeay. Is it possible to get a list of all the perl modules required by memCron? That way I can ask them to install any and all missing modules at once. Thanks!
Hi Xerotone, it’s interesting that you mention Crypt::SSLeay is required. As you can see, memCron does not explicitly use the Crypt::SSLeay module, it is required for https support by the LWP module.
The modules used in memCron is as follow:
All these modules should come as default on all DreamHost servers.
Hi again Yaosan,
I’ve got another question for ya,
Say I wanted to use memcron just for the pretty graphs, and not have it actually adjust my PS’s memory. What settings would I need to change in config.cfg to do this?
Thanks again,
Keep up the great work,
-BassKozz
Right now there’s no easy way to do that, unfortunately. If you really want to, go into
memcron.pland find the line that saysmy $time(line 208), then go to the next line and add:$set = 0;Now that I think of it, Couldn’t I just set $max_memory & $min_memory to the same number ?
er, would that not work?
That could actually work, a very interesting use of the configurable value.
If my code is robust enough (I hope! 
) then doing that should keep your memory constantly set at the value you wanted without using any of your daily change limit.
Let me know if that works out for you!
It WORKS
!
) w/out having memcron adjust it inbetween/during tests.
The reason I wanted to have memcron operate like this is because dreamhost gave me the first week free, so I want to do some load testing on my new PS and be able to track it (using the Pretty Charts
Thanks for the feedback, I should’ve thought of that first.
Yet Another Question for ya Yaosan,
Are you getting sick of me yet? Just Kidding, that was not the question…
Can memcron be used to graph and adjust the memory on a MySQL PS?
If so is there anything I should know about the way the config.cfg should be setup or is it all the same?
Thanks,
-BassKozz
I don’t have a MySQL PS so I don’t really know how it works. If it works just like the regular PS, chances are the regular installation will work just fine on that server.
You need to have shell access and writable directories on your MySQL PS otherwise memCron won’t work. You might also need to create a symbolic link that points to the csv file generated on your MySQL PS to have the graph working.
I contacted Dreamhost’s support team, and apparently they don’t allow you to SSH (shell) into a MySQLPS, so that pretty much kills the deal as far as running memcron on a MySQLPS.
Bummer
Maybe they’ll add this in the future, but for now I am going to steer clear of the MySQLPS because of this.
Excellent Job BRAVOO!!
How can I avoid seeing Internal Server Errors? I’m not sure what value to set the $mem_free_to_used_ratio variable to. Any advice would be appreciated.
You can increase the
$mem_free_to_used_ratioto reserve more memory to prevent internal server error, which happens when you run out of memory.You can also set
$min_memoryto prevent memCron from setting a memory value that is too low for your site. This could be useful if you anticipate the usage to be higher than what memCron predicts.Thanks but I was hoping for an actual example, what would a higher value for the ratio be? I’m used to seeing a ratio represented like 6:5 or something similar, not just as a single float like 1.2.
Someone PLEASE correct me if I am wrong, but I believe the $mem_free_to_used_ratio is a percentage. So 1.2 is 120% meaning, there should be 20% free above used.
So for example:
If you have 100mb used then free should be 20mb total would be 120mb.
Take this with a grain of salt because I am not 100% sure on this.
HTH,
-BassKozz
BassKozz, you’re not quite right there.
As written on the usage page:
What this means is if you are using 100MB of memory, and you want to have 120MB of free memory, then:
free memory / used memory = 120MB / 100MB = 1.2 = $mem_free_to_used_ratio
As you can see, the higher you set this value, the more free memory you will be reserving. The total memory is just used memory + free memory, and you pay for the total memory you used. On the graphing page, the total memory is actually the “guaranteed” portion of memory DreamHost provides. Add in the burstable portion i.e. swap memory and you have the real total memory usable.
Back to the question, if you would like to have less internal server error, increase the ratio to 1.6 or even 2. You will notice the difference on the graph after you apply the changes and wait for the new data points to show up.
Thanks!
Couple of minor suggestions for a future revision…
1. It would be nice if the index.php had a favicon, I usually have multiple tabs open on my browser and it’s hard to pick out which tab I am looking for unless I have a favicon to reference.
2. It would be nice if the index.php auto refreshed. But it would be even better if the auto refresh was based upon the $cron_interval variable (I currently have my $cron_interval set to 2, and I manually edited my crontab to allow for that), this way it would automatically refresh as new data appeared. Also if this was a new variable (say $auto_refresh, 1 = on 0=off) so that this feature could be turned on or off automatically.
Just some idea’s for thought.
Thanks again for all your help and the great work.
-BassKozz
Great work! Really like its and I look forward to any future development you do on it. One simple thing I would love to see would be a timezone offset. So the timezone could be matched to my current timezone, or even better would be to the timezone of the viewing user. Its minor but would be a nice touch.
Hi Jeffrey, right now the timezone offset is set through a variable found in
index.php. I’ll try and see if I could somehow come out with a way to have the timezone automatically detected for the viewing user using JavaScript, that will be really neat for sure!I’ve got another one (suggestion) for ya,
The
$mem_threshold&$emailvariables are brilliant, but I would also like to be notified if/when my memory is resized back (BELOW) my$mem_threshold.The reason being is that as the code is now we get an email if the memory is resized above the threshold, but we have no way of knowing if/when it has been resized again back to normal levels (BELOW) the threshold. So for example, I recently got a notification that I was resized above the threshold at 1am, so when I woke up the next morning I had to immediately run to my computer and logon to my memcron site (index.php) or my Dreamhost panel to check and make sure that it was back within normal operating levels, to make sure I wasn’t getting charged big $$$ for the excess memory, and ensure that everything was operating as normal. A simple email notification would have saved me from having to do that.
-BassKozz
Once again, a very good suggestion, BazzKozz. It makes sense to have the notification works just the way you described. I’ll see if I can make it work that way in the next version.
Thanks for your suggestion!
Hi Yaosan,
Thanks for developing this great tool. I’ve been using it for a bout two weeks now, and I’ve noticed that no matter what I set the ratio to, a single measured increase in usage shows a large increase in allocated memory.
http://img269.imageshack.us/img269/6669/memcronexample.png
In the example above, for the first half, my ratio was set to 1.05, and as you can see it appears to me that the adjusted memory increases more than necessary for a temporary peak.
What do you think? Am I misinterpreting the graph?
Hi Ben, the reason for the increase in allocated memory is due to the spike your server experienced, which probably increased the standard deviation significantly. This in turn causes memCron to reserve more memory in anticipation of another spike.
You can reduce the effect of single spike by either increasing or decreasing the
$num_datavalue. By setting it to a lower value, you reduce the length of time it takes for the spike effect to be completely eliminated. For example, with the default value of 80, the spike will affect the standard deviation for 7 hours. By setting it to a larger value, you can also reduces the spike effect because the majority of normal data point will outweigh the spike effect.Another value you can play with is
$mem_free_confidence. Lowering it will reduce the effect of standard deviation on memory tolerance, and hence less memory will be reserved.Got another screen cap for you, Yaosan. http://img189.imageshack.us/img189/7986/memcronexample2.png
Here are my settings:
$mem_free_confidence = 0.20;
$mem_free_to_used_ratio = 1.20;
$downsize_resistance = 2;
$num_data = 36;
So first, I don’t see why the zig zag is happening on the most current part of the graph. The average usage is not increasing so why does it jump back up?
Second, as you can see I have many approxamately 10% increases in the memory used, but then they drop back down. As long as this doesn’t exceed the memory I’ve allocated, I don’t want memcron to add more because more than likely, it will just come back down. And, Dreamhost lets you burst above your allocated memory anyway.
Third, if you look at the total graph, you can see that if the allocated memory was set at around 1GB, then the used memory would sometimes touch it, but then would drop back down. This is how I hope we can get memcron to work. …then as less memory was required overnight, it could drop down to around 750MB as well. This is my goal — to better track the minimum memory required.
So what do you think, are there different settings I should be using, or can you think of a modification to the code that might help?
Cheers,
Ben
Hello,
Is it possible to add a function to set my PS at max (4000 MB) from 7pm to midnight everyday ?
Thanks a lot !
First of all, are you doing this because memCron isn’t able to scale quickly enough for your server’s need? If that is the case, please let me know more about your server’s usage history and I’ll see if memCron can be improved to accommodate the known spike you are experiencing daily.
Here’s a possible solution for your question. First, copy your existing
config.cfgand name themconfig1.cfgandconfig2.cfg. Leaveconfig1.cfgalone and modifyconfig2.cfgsuch that the max and min memory are both set to 4000.Next, set up a cron job that runs at 7pm daily that overwrites your
config.cfg, e.g.cp ~user/example.com/memcron/config2.cfg ~user/example.com/memcron/config.cfgThen, set up another cron job that runs daily at midnight that revert your config files e.g.
cp ~user/example.com/memcron/config1.cfg ~user/example.com/memcron/config.cfgLet me know if this does the trick for you.
I installed it and keep getting the error
Warning: fopen(/home/dangerlarson/cparedirect.com/memcron/logs/ps11041.csv) [function.fopen]: failed to open stream: No such file or directory in /home/dangerlarson/cparedirect.com/memcron/index.php on line 57
Can’t open log file!
I checked & there is no file by that name in what I have on the server or what I downloaded. Is that supposed to be created on the fly & if so, any ideas on what I did wrong. Installation was pretty simple so not sure how I botched that one
Matt,
You missed the most important step of the install:
;-D
Memcron needs time to collect the data, it first collects the data in a *.log file then parses it over to *.csv. Check back at your site in about an hour, you should be good to go.
-BassKozz
Hi Matt, I think BazzKozz pretty much answered your question. (thanks BazzKozz!) Let me know if you have more installation issues. Good luck!
hey Basskozz – actually, my memory is fine. It had been about 3 hours before I emailed you and now it’s been about 7. Still the same error.
Who said anything about your ‘memory’ ?
Anyways, couple of things aren’t adding up here, your memcron dir/site: http://www.cparedirect.com/memcron/index.php
Currently reads:
But line #57 of index.php doesn’t have anything to do with the log/csv files, it should be referring to line #85 (
$fp = fopen($log, 'r') or die ("Can't open log file!");
) for this particular error. WeirdCouple of questions:
What version of memcron are you using?
Have you modified the index.php file in any way?
Did you make sure to set the correct
$key&$psin the config.cfg file?- using .3 – will try .4
- have not modified the index.php
- $key was copy/pasted right from the dreamhost api screen
- $ps came from everything before the “.” in my host name “psXXXXX”.dreamhost.com –
- btw, was referring to memory in my head, not 1’s & o’s memory.
Thanks for the great tool, Yaosan. I’ve already donated.
Question: in my notification e-mails the running processes section is coming in blank. Something I’m doing wrong?
Hi Jamon, things like that do happen when your server gets very busy, say it is approaching its memory limit or the CPU load gets a little higher than normal for the shell to respond properly. This is also the reason why you will sometimes see a couple of missing data on the graph. Sometimes, the command just didn’t get executed perfectly, and many times you can’t really tell for sure why it happens.
I assume you only experience that missing running processes section once in a while. If it happens consistently, let me know and I’ll look into it.
Thanks for the donation btw!
It’s definitely every time–I haven’t had one come in with any info yet. Here’s an example:
Date: Tue Jul 7 06:21:02 PDT 2009
Total: 300 Used: 276 Free: 24
Mean: 112 Stdev: 14 Load: 0.00
Mem: 150 Swap: 150
Tolerance: 22 Tolerance (free): 363
Target: 320
Memory was resized from 150 MB to 320 MB!
-----
List of running processes:
This report is generated by memCron at 06:21:02 AM PDT (2009-07-07).
Okay if it happens every time that something must be wrong. FYI, memCron is trying to run the following command to generate the list of running processes:
top -b -n 1Try logging into your server and type this command in shell. If this works then I can’t see any reason why it returns nothing when memCron tries to run it.
Hm, yeah, from a shell it is working–it returns the list of processes just fine.
I tried this though and it didn’t work:
User:
It didn’t post:
exec("top -b -n 1", $a);
echo "User: " . exec("whoami") . "";
print_r($a);
…hopefully that came through.
Jamon, try running this line in your shell:
perl -e '$i =`top -b -n 1`; print $i;', if this works then the process list should work. If not then something is wrong with your shell or perl.Thanks, Yaosan. It does work and I got a list of running processes. But I just got an e-mail where the list of running processes was blank again. Somehow it’s not transferring to the e-mail.
Any other ideas?
No unfortunately. One other thing that memCron does differently compared to the code I showed you is memCron will chomp the “top” output. I don’t see how this can make a difference. Sorry, I’m out of idea.
Thanks anyway, Yaosan. I’ll play around with it some more and if I get it to work I’ll post my solution here.
Matt,
I see your site (http://www.cparedirect.com/memcron/index.php) has changed:
The problem is it’s trying to reference a log file called “ps11041.dreamhost.com.csv” but that’s not correct. The file should be “ps11041.csv”. Make sure your config.cfg file reads:
$ps = "ps11041";Also go into your cronjob (DreamhostPanel>Goodies>CronJobs) and enable email notifications (email output to:), and then post back here any errors that get sent to you via email. Maybe that might shed some light on the issue.
HTH
p.s. I know which ‘memory’ you were talking about, I never said anything about your memory (in your head or otherwise). I was just kidding around with you when I said you missed a step… Sometimes it’s easy to speed read install instructions, and I thought thats what you did.
The .dreamhost.com was something I just added this AM. previously just had the PSXXXX in there and it didn’t work.
Removed it & here is the error I’m getting via email.
Uncaught exception from user code:
Cannot open memory log file! at /home/dangerlarson/cparedirect.com/memcron/memcron.pl line 119.
at /home/dangerlarson/cparedirect.com/memcron/memcron.pl line 119
ideas?
I assume it is probably a UNIX ‘permissions’ problem, but I can’t be sure.
Can you logon to your shell (SSH) and post the output of the following:
ls -la /home/dangerlarson/cparedirect.com/logs/Otherwise I am stumped, Yaosan ?
I do think it’s a permission issue. Matt, you can try chmod your memCron’s log directory to 775 just so to ensure the cron running user have permission to create the log file for you.
Also, make sure you set up your cron job properly by picking the correct shell user that have access to the memCron directory you uploaded.
Hey guys, the 775 took care of it. Thanks for the help
Told you Yaosan
correction, post the output of this:
ls -la /home/dangerlarson/cparedirect.com/memcron/logs/Yet another suggestion for ya Yaosan,
It would nice if you kept a log of the
targetfor graphing.I am using the “Downsize count” in my graphs to keep track of when I might expect a downsize, but it would be nice if we could also see what the
targetis for the downsize, currently the only way to see thetargetis to enable MAILTO in the crontab. Also it would be nice if we could have multiple lines in the graph instead of just two (3 if your using Used+Total Memory). My dream graph would consist of the following:Just some suggestions. Are you getting sick of them yet?
The graphing part can still be improved, no doubt. Initially I want to make all the data selectable via checkboxes for plotting just so you can instantly turn any data on and off. However, since all data has different min, max value and units (CPU uses load while memory uses MB), it’s very difficult to set up the axis right. It’s probably impossible to have used memory, CPU load and downsize count all on the same graph since they all have different units and we have only 2 y-axis usable.
I can add target to the csv file just so you can pick it from the drop down field on the graphing page in upcoming versions, does that help at all or you’re still stuck with your dream graph?
No that would be great if you could add
targetto the graph drop down list… I can switch back and forth between the different graphs, or maybe just have 2 graphs open at the same time in different firefox tabs.I can’t always get what I want ;-P
Thanks for hearing my suggestions
What about having the Y axis be a % of max? For example, memory would be % of the $max_memory, CPU load would be % of some max load…etc…
I can see the problems with this suggestion as well.
One thing that could be a little confusing is when you select two memory Y axis.
Here’s the query string:
/?y1=mem_total&y2=mem_free&n=60
The left and right axis are a different scale, making it difficult to compare the two. Maybe if the two Y axis are both MB they should have the same scale (min and max).
The reason why the 2 y-axis are not using the same scale is because the fluctuation for one (e.g. stdev) can be so small that when you use the same scale as the other axis (e.g. total memory), you won’t be able to see the small fluctuations and it will look close to a straight line to you.
Sweet tool, been running it for a while now on my PS. Any chance you can also log (and graph) the number of connections (netstat -nt) as this will help as a guide to understanding why we are getting spikes in memory usage .
I’ll second that, it would be great if this were attached to the
$mem_thresholdemail.Good idea ozgreg, I’ll probably add 2 versions of netstat output, one with IP and another with hostname into the next version. Is there a way to get both IP and hostname using a single netstat command?
fyi I’ll be away for about 2 weeks so there won’t be any response and updates from me during this period.
Memcron was working for me before, not sure what happened. Before I had the nice graphs that showed how memcron was adjusting my memory. Now on the graph it appears memcron is doing no memory adjustment though the cron emails claim some adjustment is being done.
And I’m gettting this in email. (note: I changed the URL and X’d out the values in this comment post.)
sh: ps10379: No such file or directory
sh: ps10379: No such file or directory
Date: Tue Jul 21 00:31:10 PDT 2009
Total: 700 Used: 495 Free: 205
Mem: 350 Swap: 350
Mean: 270 Stdev: 68 Load: 0.26
Tolerance: 129 Tolerance (free): 513
Target: 558
Memory was resized from 350 MB to 558 MB!
Uncaught exception from user code:
Error at https://panel.dreamhost.com/api/?key=&cmd=dreamhost_ps-set_size&unique_id=8081e7d3-b31c-48ab-a84a-2c7cd02239cb
&ps=&size=558
401 Not Authorized
Aborting at /home/hawaii247org/hawaii247.org/memcron/memcron.pl line 272.
at /home/hawaii247org/hawaii247.org/memcron/memcron.pl line 272
Guessing from your 401 error that your API key is incorrect, suggest you might want to verify the DH API key again (or generate a new one)..
I have a MySQL PS and I would like to try to get this working on it as well. Since you can’t SSH to the MySQL PS, is it possible to monitor the current memory usage/load via the DH API instead and react to that instead? I’m willing to test this solution if you like.
This is a tricky proposition, because that would require giving memCron the user&pass to your dreamhost Panel, which is a security concern. As it stands right now memCron only has access to the API and can only run API calls that you’ve enabled when you create the key. Allowing memCron to access the Panel is a big no-no as far a security is concerned because with that information someone could do some real damage to your sites/domains/etc…
While the config.cfg file is protected with .htaccess (i.e. you can’t view the file from http://memcron.domain.com/config.cfg) if the .htaccess file were to be mistakenly deleted this would open you up to the possibility of someone gaining access to your DH Panel’s user&pass. :O
Not only that but memCron is currently only setup to handle SSH commands not webscraping, so it would be a MAJOR undertaking to scrape the DH Panel, and if DH ever changed the way they reported information on their Panel it would break the whole thing… Not worth the effort IMO.
A possible alternative would be to use the api’s [code]dreamhost_ps-list_usage[/code] call to determine the MySQL PS’s usage, it is unclear how often this information is updated, but from what I can gather from the wiki: http://wiki.dreamhost.com/Api#dreamhost_ps-list_usage
This gives the results of the past 30days in a daily AVERAGE of the amount of memory used. Therefore it might be possible to setup memCron to run on a “daily” basis to check the previous day’s average and adjust accordingly. Not that great of a solution compared to what memCron is currently setup to do (run every 5minutes and adjust accordingly), but I think it’s the best alternative available at this time.
I would submit a request to Dreamhost and ask them to change the [code]dreamhost_ps-list_usage[/code] call to return the CURRENT usage statistics instead of a daily average. Maybe if enough of us request that as a feature DH might make a change.
If there’s a way to get say 5 minute average usage via API then it shouldn’t be hard to adapt memCron for MySQL PS. However, I do not have an MySQL PS myself so I might need a demo access to test the code out. Anyway, it will probably be some time before this API feature gets implemented, if ever.
I’ve had memCron set up on my website for over 24 hours now, but it appears to be failing to generate the correct files in /logs/, as can been seen here: http://iiichan.net/stuff/memcron/
Any ideas?
David,
Two things:
1. Can you logon to your PS via SSH and run paste in the following:
ls -la ~/iiichan.net/stuff/memcron/logs/Then paste the output here.
2. Go into your DreamHost Panel>Goodies>Cron Jobs, and modify the memCron cronjob and put in your email address in the
Email output to:field, and report back any errors you get via email.-BassKozz
total 12drwxrwxr-x 2 humblefool pg198835 4096 Jul 22 17:22 .
drwxrwxr-x 6 humblefool pg198835 4096 Jul 22 18:48 ..
-rw-r--r-- 1 humblefool pg198835 49 Jul 22 18:48 .htaccess
I already turned on email reporting for the cron job, and haven’t received any emails.
running ‘perl /home/username/iiichan.net/stuff/memcron/memcron.pl’ generates a file and the correct memcount, but the cronjob doesn’t seem to be functioning. Hmm.
Figured it out. Cron wasn’t running. A quick reboot fixed that.
Yeah, if you aren’t getting any emails AT ALL, and you have the
Email output to:set, that means the CronJob isn’t running.Would it be possible to modify the program so that it could break down the chart display to show which domain the CPU and memory load is coming from? I have several domains running on the same PS and I would love to be able to chart the usage for each domain rather than just the overall?
If this were possible and were built into memcron I’ll donate again immediately. That would be so helpful. Not sure if it’s possible though.
I highly doubt that this is possible, as the processes that are running on the PS don’t differentiate between domains (or subdomains).
Yaosan might know?
We could probably track per shell user, though. And if you have one domain per user then it’s easy to tie them in.
Good point… Didn’t think of that.
That might do the trick however, if your using mod_php, this won’t work as all the processes are run under the user “dhapache”. Not sure if the same holds true for standard PHP, as I haven’t tested it.
Also there are some other processes that run under:
root, postfix, dhapache, daemon, etc…
That wouldn’t be able to be directly tied to a specific domain (or subdomain).
True. But those could still be listed separately and you could debug from there. I’d definitely like this functionality.
Jamon,
Don’t get me wrong, I am still interested in seeing this as a possible addition to memCron in the future.
Sure, BassKozz, I think it would help a lot of us. My PS has taken an average 100 MB jump in the last 10 days for no apparent reason and I want to find out why. I don’t know what was active in the
topcommand before the jump so I can’t compare very well. Rebooting has no apparent effect.It won’t be that easy to track usage by shell user. The only way I can think of is parsing the top output and average out the usage over say one minute for each user, and use that to assume how much memory is each user using.
It will take some testing to just get that data properly stored in csv and plotted using PHP and Javascript.
I installed memcron about a weeks ago. It has been working relatively well so far. A nice side effect of the graph is that it allowed me to catch a problem with the cron jobs. I’m really looking forward to see a 1.0 version.
It also helped me catch a problem…
For some reason on 7/25 my cronjobs stopped dead in their tracks, I tried to “crontab -l” and got a “permission denied” error. Long story short my permissions get fudged up some how. I contacted DH support and they fixed it, but they couldn’t explain why this happened in the first place
Anyways, I am curious to hear what problem you ran into Joe?
I don’t know what the problem was as I found out a couple hours after things went back to normal. I contacted DH support and while they saw that something went wrong, they couldn’t figure out why…
I agree that the graph is very useful to find out if your server is down, which will show up as missing data points for a period of time.
From the graph, I also notice at around 9:15pm EST yesterday, extra swap memory is allocated to your server for the same amount of memory set via API. I believe DreamHost might be rolling out an update to allocate extra swap memory to handle spikes better for customer that is not running memCron and is having trouble with internal server error due to spikes.
As well all know, memCron is already doing a great job without the extra swap memory. If this update is permanent, memCron can operate at a lower margin, and you will be able to save even more money with it.
Very good application. thank you!
Bug report: memCron appears to be ignoring the $max_memory variable; it’s set at 1500, but as you can see from the graph (http://iiichan.net/stuff/memcron/), that doesn’t appear to be holding true.
that’s http://iiichan.net/stuff/memcron/ of course.
I think DreamHost is rolling out a new update that gives extra swap memory to your PS server, this is the reason why you see the jump right after memCron changes your memory setting, signifying more memory is available than what memCron expects.
If this DreamHost PS change is permanent, I will update memCron to adapt to the new memory settings.
I also notice you can now create a root user for your PS via DreamHost panel.
I’m not a Perl programmer (PHP only) but I noticed in memcron.pl on line ~339 it says:
} elsif ( $mem_target > $max_memory ) {
Shouldn’t it be:
} elseif ( $mem_target > $max_memory ) {
? That’s probably the issue.
No, “elsif” is proper perl syntax.
Oh…
okay. Perl seems a bit cryptic at times compared to PHP.
Unfortunately, this doesn’t work in Firefox 3.6a1pre. The graph is not shown, the text is not styled and in the error console, it says “Error: $.plot is not a function; Line: 158″
I don’t know whether the problem is memcron or Firefox. Could you please have a look at that? Thank you very much!
Forget it, Adblock Plus was the problem! (my filter blocks stats.mydomain.com
)
I’m getting a lot of memory spikes and plateaus.
http://www.devcsd.com/memcron/
What do you suggest? I changed a few things to combat it less than an hour ago but it’s still doing this.
$max_memory = 600;
$min_memory = 250;
$mem_free_confidence = 0.992; // was 0.999
$mem_free_to_used_ratio = 1.2; // was 1.5
$downsize_resistance = 2; // was 6
having the same problems here- this just started this morning. memcron seems to be upsizing too much…
Same here, started at roughly 1:30am EST for me.
I did some testing and found an Odd result… I set
$max_memoryand$min_memoryto 150MB. Yet I am still getting these results:http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-bizarre.png
http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-bizarre2.png
http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-bizarre3.png
Also according to my DH Panel, memCron keeps changing the memory size, to 150MB, even thou it’s already there:
Seems like a Serious problem with memCron and/or the DreamHost API.
As Yaosan said above, DH is probably rolling out some changes which has caused memCron to go haywire. I’ve disabled memCron on my PS until this gets straightened out.
Since this all just started at the same time for everyone, it looks like it’s on Dreamhost’s end.
Something strage is heappening with my memcron: In the graphs shows that I have 225Mb of memory, when I actually have only 150Mb.
Then I’ve manually change to 188Mb and now the graph shows 280Mb
Something is not OK, I’m gonna reinstall it. Just to let you know of this problem.
Please support memCron of MySQL PS servers. Thanks!
We had this discussion before, and the conclusion is it is not possible for memCron to run on MySQL PS unless DreamHost starts providing API to access instantaneous memory usage on MySQL PS.
For anyone who has reported an issue due to the recent DreamHost updates, please give memCron 0.4.1 a try and see if it fixes your problems.