First of all, I want to say thanks to Ben for his relentless support helping other users with their issues and constantly providing updates to the memCron code base due to some of the recent DreamHost PS changes. After spending the weekend polishing up the improvements contributed by Ben and Bret, memCron 0.6 is finally here.
Here’s a list of notable changes:
- Removed dependency on modules with built-ins and system utilities, including SendMail, LWP etc.
- Get memory info from /proc/meminfo
- Get current memory size from API rather than OS
- New option to disable updates
- New options to send out notification when changing memory setting or getting API errors
- New chart formatting options: target memory, cached memory
- Now display http server name instead of actual PS name on the graph page for extra security
Download: memCron 0.6 (34.12 KB)
Please give this new version a try if you’re having issues with the older versions.
To upgrade, follow step 1-3 listed here. FYI, the old “zscore.pl” is no longer used and can be safely removed.
{ 2 trackbacks }
{ 120 comments… read them below or add one }
Awesome work guys.
Installing now!
I’m mostly commenting to say thanks (Thanks!) and to get notifications of future comments, but I’ll also take this opportunity to point people to a Dreamhost suggestion to extend the api to allow mysql ps’s to also be managed by memcron. Please give it one of your votes!
https://panel.dreamhost.com/index.cgi?tree=home.sugg&category=_all&search=extend%20web%20panel%20api
Do the list_usage and set_size calls not work for mysql ps’s? I don’t have one, so I can’t verify this myself, but it seems like they should.
The list_usage call is overkill, a get_usage call that returned just the most recent value would be much less intensive.
thanks trying this version now…
Thanks a lot, I’m giving it a try.
Thank you!
I ‘ll upgrade today
In the logs, notices and everywhere else, memcron uses the hostname where memcron runs. With $ps no longer in use within memcron itself, it should have been left as the title of the chart….
New version… http://ben.pastebin.com/yiYFEgcj
Adds HTTP::MHTTP (which must be installed with SSL enabled, see the readme in that module).
Adds the beginnings of a log to track exactly what memcron is doing.
Enjoy!
The script keeps posting these error message:
main::normsinv() called too early to check prototype at /.../memcron.pl line 94.
main::normsinv() called too early to check prototype at /.../memcron.pl line 95.
Any chance to have these fixed?
They’re just warnings, can be safely ignored — but my recent update doesn’t have the issue.
I got the same error messages, even with the recent update, and still not working…
Me too – exact same message… (version 0.6 of script)
Try my updated version of the script, it’s been working for me – see my comment from July at the bottom of the page:
http://memcron.com/2010/03/memcron-0_6/#comment-514
Thanks to Ben for keeping up with the development of memcron, and to Yaosan for finally catching up. I’m using it right now w/out any issue so far.
I’ve got a problem where memCron is not generating a log file. Everything was working with the previous updates before 0.6 but now I see that no logs are being generated. Permissions on the logs directory are set to 775, I have the cron job set in my DH panel (I even deleted the old and created a new one to see if that would work), I have waited hours for a log to generate, I have and API key set in config for all functions (even re-did that too when it wouldn’t work).
So again, I set it all up as per the upgrade instructions. No log being generated after hours. I re-did everything (re-entered API Key, PS, CHMOD, Cron job, yaddah, yaddah) still no log being generated.
Any ideas?
Hi Baron, is any files created at all in your logs directory? You might want to try running “perl memcron.pl -d” directly in shell and see what output you are getting.
I’m getting (ps exed out)
[psXXXXX]$ perl memcron.pl -d
main::normsinv() called too early to check prototype at memcron.pl line 94.
main::normsinv() called too early to check prototype at memcron.pl line 95.
Give my 0.6.1-ben version a try ( http://ben.pastebin.com/yiYFEgcj ) — it should be a little more verbose when you run “perl memcron.pl -d”.
Aloha Ben, I get this output and nothing else. And no files in logs directory. Then again it might be a DH glitch and I should shut-up as it keeps showing that I’m using zero memory so I’m being charged the base price when I look in my DH control panel resouces. And yes, I’ve rebooted the server a few times without clearing these issues.
[psXXXXX]$ ls
config.cfg css favicon.ico images index.php js logs memcron.pl
[psXXXXX]$ perl memcron.pl -d
[Wed, 24 Mar 2010 15:24:04 -0700] memcron 0.6.1-ben started (pid: 7127)
[Wed, 24 Mar 2010 15:24:04 -0700] calling api list_ps
[Wed, 24 Mar 2010 15:24:04 -0700] using system curl
Fixed now. Apparently DH finally realized they had a problem and fixed the memory issue. http://www.dreamhoststatus.com/2010/03/26/homie-vserver02/
Thank you for keeping on with memcron
Ben, would you please include in the index.php the following before the tag?
<meta http -equiv=”refresh” content=”300″>
That will refresh the page every 5 minutes and display a most recent graph …
I always have memcron displayed on my browser…
It wont hurt any body else.
My memcron isn’t measuring the Used memory on the server. Have any you you had trouble with this?
See here: http://www.artcrete.com/memcron/index.php
I have a problem with memCron 0.6 regularly the execution of the script is interrupted with an error 500, I not found any trace of the error in the logs of the server, I have the same problem with “0.6.1-ben”.
Apparently you have the same problem on the demo: http://memcron.com/demo/
I already tried with the support of Dreamhost but do not know how to help!
Any idea?
I have the same problem as “ppalli”.
The memcron script is not executed some times.
That is visible in the graph.
The dots representing the execution time are missing.
Please see the demo: http://memcron.com/demo/
and http://www.sdtv.gr/dreamMemcron/
Regards
I’ve now upgraded one server to 0.6.1-ben, and it’s working so far. Don’t have mhttp installed yet, but in time. The only disappointment is the instability of the DreamHost API. Lots of gaps in my graph due to the API failing to get the currently allocated memory for the machine. But since their latest restructure on the way they count memory, there really isn’t any need to adjust it like there used to be, so no worries.
I seem to be getting a “No response” error with Ben’s 6.1 version. I double checked my API key, server name, and the logs directory is set to 775. Does anybody know what the cause might be?
Full output:
memcron 0.6.1-ben started (pid: 7048)
calling api list_ps
using system curl
No response
memcron 0.6.1-ben complete
Try this: http://ben.pastebin.com/1tWwpXJ5 save it in your memcron directory and run it from there like “./dh_call list_ps”
It will display the curl command (which you can then run directly) and its output.
Ben thanks for your help but I don’t understand. I saved that file as file.pl and uploaded it to my memcron folder. Then I run a command via shell? And what do I do with the results?
Ok, so you named it file.pl, so you’d run the command “./file.pl list_ps” from the shell. It should print a perl hash with your ps listed in the results. If you get nothing, that’s the equivalent of “No response” from within memcron… that could mean that your curl doesn’t work.
I tried your command to the server but I got a permission denied error.
Then I tried “perl file.pl list_ps” (without the quotes) and that gave me some output
I removed the key, private server name, and unique_id from the output:
curl -s -d format=perl -d cmd=dreamhost_ps-list_ps -d ps=removed -d unique_id=removed -d key=removed https://panel.dreamhost.com/api/
Does that give you any clues?
Permission denied means that the key you’re using doesn’t allow list_ps, which it needs to for this to work properly. That means, you’ll probably have to create a new API key with the appropriate permissions. The reason it uses list_ps is because we can no longer infer the allocated memory from what the machine itself reports internally, so the API call gets the current “total memory”.
Not in this case, permission denied means that he didn’t set the script executable. so either “chmod 755 file.pl” or run it as “perl file.pl list_ps”
Ben, I ran the 2nd command and got the output listed in my last post. What do I do with the output?
Dreamhost moved my PS a couple days ago and the script started working again. Not sure what happened but glad to have the script back.
I’m getting a “No response” error with Ben’s 6.1 version. Any Ideas?
Full output:
memcron 0.6.1-ben started (pid: 3711)
calling api list_ps
using system curl
No response
memcron 0.6.1-ben complete
I’m running 0.6.1-ben, and am finding that it is too reactive. For an example see this chart. The 180MB peak usage would have cause memcron to push my allocated ram to 459MB, while 300MB was plenty adequate to handle it. I can see it being a bit more reactive if it were say within 10 or 15% of allocated memory, but there was still 40% left. I’m not sure how to adjust it to make is less reactive to artificial “needs”. And since there doesn’t seem to be any need to adjust memory at present, I have updates disabled which saved it from an hour or two of extra cost for this round. Just thought I’d give an example of something that isn’t quite what it needs to be.
Thanks for memCron. Just found it and it’s running great on my PS. In fact, due to it I just found out that the control panel stats were way out and I’d had my memory set too low for some time now, causing slowness and agitated hosting clients, so thanks very much for the wondrous memCron.
One request for 1.0- could we please have an option to set our timezone so the time values mean something to us non-East-Coast-USAers? In the meantime, is there a hack that I can do to achieve this?
C*
in index.php, there’s a $gmt_offset variable you can change. It definitely would be nice if you could set a timezone in the config file though so it would adjust daylight time and all. But at least it’s hackable.
Ah great- sorry, I should have looked harder. Adjusted and working well. Thanks for your help.
Anyone else getting http error 500 on the list_ps API call?
I do occasionally, but most of the time it seems to work fine. Did you double-check your API key and machine name?
I’ve been running memCron for a few weeks now and although it’s running fine, I’m having a few issues fine tuning it.
I’ve seen the examples here on the Screenshots page and they show memCron tracking the Total Memory fairly close to the Used Memory which is what I’m trying to achieve. Memcron is working as you can see here:
http://marcushill.com/memcron/
but it’s leaving a lot more headroom than I am trying to attain. I’ve yet to see the cached memory being used at all. I know ideally I wouldn’t use any cached memory, but it is available for emergency spikes and I’m expecting to see it used periodically but have yet to.
I initially ran with the default settings for a few days, but wasn’t getting what I wanted. I’ve tried various settings in config.php these are the relevant settings as I have them set currently:
I’ve read the Usage/ FAQ but still don’t really understand what the Minimum free vs. used memory ratio and Confidence level settings mean. It would be great to see some examples to illustrate the effect these settings have.
Can you see any problems with my settings and ideally recommend some?
No emails have been received, even though both the threshold is set to 650MB, change & error options are set to 1 and it’s set to send to a valid/working DH hosted address on the same domain as memCron is running.
I also tried running the script from the command line as shown in the Screenshot and receive the following error:
perl marcushill.com/memcron/memcron.pl
main::normsinv() called too early to check prototype at marcushill.com/memcron/memcron.pl line 94.
main::normsinv() called too early to check prototype at marcushill.com/memcron/memcron.pl line 95.
TIA
weird mine stopped adjusting for the last 1-2 days…and i didnt change any settings.
I’m having the same problem where memcron seems to have stopped working for over a week now. I re-installed memcron 0.6.1, re-checked the cron job, created a new API key with all functions and although it’s generating a readout it’s still not adjusting.
http://www.hawaii247.com/1memcron/
Also why is it that both memcron and the Manage Resources graphs so far off from what I’m seeing when I ssh into my server? When in terminal it shows my memory usage constantly above 300MB but memcron shows it only less than 200MB of usage? The graph in web panel mostly matches the memcron graph.
I assume the info via top -c via SSH is accurate of the situation but why so far off? And it doesn’t seem like memcron can react fast enough to memory needs if that’s the case.
I can only guess as to why they are so different, but it seems to me that they started exclude Apache’s memory allocations from the total seen in the graph. I can’t say it makes me unhappy because prior to their last round of wild adjustments I was unable to keep the memory stable enough to keep the machine from crashing regularly. And since most of my sites simply serve static pages, it was always Apache causing me the issues. Since the last changes, the memory usage rarely exceeds 100MB so memcron is almost unneeded for me now. But I like having something besides the DreamHost graph tracking my actual memory usage, so I keep it on.
In SSH right now I see::
top -c: 275 MB used
free -m: 255 MB used
/proc/meminfo: 288 MB used
I don’t see how it adds up because memcron and the DreamHost graph both report around 80 MB and I copied the calculation out of memcron for /proc/meminfo and got 288 MB in the same timeframe…. sigh. I don’t pretend to understand it, but I’m glad it works better than it used to.
Oh, and since 300MB is now the minimum allocation, there isn’t any need to adjust memory. That’s why your’s isn’t changing.
Okay, a couple things.
I cleared-up memcron not working. For some reason the directory permissions were wrong so I changed logs directory 775 and I think memcron is now working again.
As for why the discrepency in RAM usage I was pointed to this article:
http://www.linuxatemyram.com/
Apparently DreamHost isn’t taking into account cache RAM being used which is why viewing free RAM from terminal always looks worse than via memcron or Web Panel.
Well, I spoke too soon. memcron is NOT adjusting my memory
And this after a couple hours of data points being recored and my memory usage running about 100MB. My memory is stuck at 400MB where I had put the slider before. In my memcron config file I have the lower limit set to 300.
same. i had manually adjusted the memory to 2000mbs to test something and then it stayed there until i had to manually adjust it again.
Hey folks, here’s what’s going on : (I heard back from DH Support)::
*paste*
Yeah, this is because we changed our whole memory set up, we no longer
use swap so memory is read different. Check this site out to get a
better idea:
http://www.linuxatemyram.com/
Basically if the developers don’t contact us if they are having any
issues on our end, not much we can do. They just need to rework their
script (if they decide to) to work with our new way of reporting memory.
If you have any other questions or need anything else, just let me know!
Thanks!
—————————-
Looks like we need a whole revamped version of memcron.
S
memcron 0.6 IS the revamped version, and the memory usage it reports in its graph should be very close to what you see in DreamHost’s graph in “Manage Resources”. If it’s not adjusting memory when it should, make sure you done have this in your config file:
# Disable memory updates?
$disable_updates = 1;
It should default to 0 so that updates continue, but if you set it to 1, it will never updates. It’s a great option because you can now run memcron for a while to monitor what it thinks memory should be before allowing it to manage it fully. Just watch target memory.
If it’s still not updating when that’s set to 0, then something else is wrong.
Hello,
I’m still having problems with memcron at DH. First, while it has been working, it was great and for that, I’m really thankful, you guys rock!
Well, I was using 0.41, when a couple days ago it has stopped working. I updated to 0.6, just replacing the files but memcron stopped logging and therefore changing the memory.
I updated to ben’s 0.61, with no change.
Then decided to do a brand new installation: I got a new API, deleted all logs and started a brand new memcron instalation. But still, memcron hasn’t been able to log, even with all files CHMOD 775 – not just the logs directory, it doesn’t log! So you can see this error log:
Warning: fopen(/home/.kitty/dentvii/vejaisso.com/memcron/logs/vejaisso.csv) [function.fopen]: failed to open stream: No such file or directory in/home/.kitty/dentvii/vejaisso.com/memcron/index.php on line 94
Can’t open log file!
Now, I if you check vejaisso.com/memcron you will be able to see my old log, where it stopped logging at 11:10.
Thanks for any help at all.
Is memcron.pl marked executable?
What happens when you run
perl /home/.kitty/dentvii/vejaisso.com/memcron/memcron.pl
?
Yes, it is marked executable.
It runs, no error message, but doesn’t log nor creates log files if there are none.
Are you sure it’s being run from cron? Whose crontab do you have it configured to run from?
It is the same crontab as it was before, so it is running. The user is the main one, Shell enabled user.
Getting this error for my memcron:
[Tue, 18 May 2010 15:55:02 -0700] memcron 0.6.1-ben started (pid: 31593)
[Tue, 18 May 2010 15:55:02 -0700] calling api list_ps
[Tue, 18 May 2010 15:55:02 -0700] using system curl
[Tue, 18 May 2010 15:55:02 -0700] No response
[Tue, 18 May 2010 15:55:02 -0700] memcron 0.6.1-ben complete
/usr/local/bin/setlock -n /tmp/cronlock.1484602.76386 sh -c $’perl \176dentvii/vejaisso.com/memcron/memcron.pl -d’
Actually, my problem resembles a lot Baron’s problem. I’ve contacted DH regarding this matter. Let’s see.
The script stopped working for me as well. I did a full reinstall of memCron + bensCron but no dice. The error I’m getting is “Error calling set_size: unique_id_already_used.” Any ideas?
Same for me, too
Now memcron didn’t change the memory size.
Debugged it to my email and got:
main::normsinv() called too early to check prototype at /home/xxxx/xxxx.xxxx.com/memcron.pl line 94.
main::normsinv() called too early to check prototype at /home/xxxx/xxxx.xxxx.com/memcron.pl line 95.
Unable to open sendmail: No such file or directory at /home/xxxx/xxxx.xxxx.com/memcron.pl line 288.
I’m getting the exact same error as Erwin K … anyone have a fix yet? Version 0.4 seemed to work without any issues. What happened???
Is there any news on this? It appears my script stopped working as well. It sucks having to worry so much about the ram usage.
Anyone have any ideas? Is there a new version being developed?
I’m giving up on memcron for now. Going to try Dreamhost PS Manager
http://www.7is7.com/software/dreamhost/psmanager.html
I’ll let you know how it goes.
Sorry Baron (and everyone else)! The project I’m using the PS for has stalled, so my server is sitting nearly idle with no need for resource management. Like a lot (some?) open source developers, my development on memcron was entirely selfish
In the end, this is Yaosan’s baby! But, I hope to resume my project and will have a personal need to work on this again soon.
As with Ben, my work on this was to support things I needed. With DH’s last changes to memory management, my usage went from 700-800MB average to around 75MB average. So while I still use it to monitor memory, there hasn’t been any need for adjustments. What might help track this down, is to set $log_level=5; in your config.cfg file and post the last section of memcron.log along with memcron’s output. You can get the output by either running it from the command line, or by having cron email you the output. I’d prefer having cron do it so you know it’s running the same way. You can always turn it right back off once you have one that fails to set memory. If it’s failing to set the memory, there must be a reason for it. You might also want $alert_on_error=1; so that you get an email notice when the API fails.
I’ve been using Dreamhost PS Manager and that’s been going well but still wanted to have memcron active so I’d have a nice graph to view. But I’m getting an error in my logs. I’m constantly getting:
[Sat May 29 17:30:31 2010] [error] [client 176.204.107.90] client denied by server configuration: /home/username/hawaii247.com/memcron/index.pl
Also there was another error where memcron could not find the file loadingAnimation.gif in the memcron images folder so I put one in there to get rid of that error. I’m guessing either that part of the code fetching that graphic needs to be deleted or the image needs to be included in the zip file.
Still getting this error in my logs:
[Tue Jun 01 00:03:44 2010] [error] [client 176.204.107.90] client denied by server configuration: /home/username/hawaii247.com/1memcron/index.pl, referer: http://www.hawaii247.com/1memcron/index.php
Other than that I’ve been able to troubleshoot what I believe were three things burning up my memory:
1. I had switched to W3 Total Cache. I liked it and its features but I read one place online that it can use a lot of memory because Dreamhost uses NFS for its file system protocol. I’ve switched back to Hyper Cache and that reduced memory usage. (I may switch back to W3TC in the future).
2. I had some old Javascript files from 3rd party sites that I pull content from so I updated the code for that and reduced the errors.
3. I switched from FastCGI to PHP CGI in the Manage Domains section of Web Panel. FastCGI was keeping too many processes open and gobbling up memory. See this tutorial: http://return-true.com/2010/05/wordpress-php-cgi-vs-php-fcgi/
So it seems that I’ve tamed my memory problems for now. The big culprit in all of this was using caching. With caching turned-off the site used less memory but I couldn’t leave it off or the site would die under a heavy load, plus rebuild time for web pages would take longer.
If anyone can figure out why the server is trying to execute a non-exisitant perl script index.pl I’d be grateful.
I have some ideias that should be interesting for many users.
1: show 2 times on the graph, the local time adjusted like is done now via timezone and include the server time, that might be showed in a little font near the local time.
2: allow more combinations of lines on the graph.
Thank you for this nice work.
News from DH about Memcron
> Hi… I have memcron and it stops adjusting memory… is there any know
> issue about memcron that you can tell me? I’m using version 0.6… and this
> is the warning message. main::normsinv() called too early to check prototype.
> This not seems to be the cause… since it was working and with no changes,
> it stoped working. It shows the graph but does not really adjuste the memory.
>
> The memory just stay on the what I set on panel.
>
>
> this is the email I receive from DH cron
>
> main::normsinv() called too early to check prototype at
> /————–/memcron/memcron.pl line 94.
> main::normsinv() called too early to check prototype at
> /——————/memcron/memcron.pl line 95.
> Total: 633 Used: 235 Free: 398
> Mean: 199 Stdev: 18.97 Load: 0.10
> Tolerance: 38 Tolerance (free): 303
>
> Target: 538
>
>
> As you can see… memcron calculates the target correctly…
>
> And there is no error reported by API usage, since memcron would report me by
> email.
> So… I believe API is may not be working…or something else is wrong at DH
> side…
>
>
> Could you help me , please?
>
Hey there! Unfortunately there isn’t much I can do to troubleshoot
Feel free to email me directly here:
memcron as it wasn’t written by us and we do not support debugging third
party scripts. I haven’t seen memcron work(or work right I suppose)
since our change in memory reporting but that is something the author of
the script needs to iron out. I apologize i can’t be of more assistance
(I did run some updates/configs just in case but I kinda doubt it will do
anything). Please let me know if you have any other questions as I’ll be
happy to help.
Thanks!
Jason Y
psmanager works flawlessly – and it would seem, better than memcron ever did. I will miss memcron pretty graphs, but this other option is actively maintained and actually works: http://www.7is7.com/software/dreamhost/psmanager.html
thanks memcron – goodbye and good luck!
At least until memcron functions again
Thanks for the heads up about psmanager. Installed it this afternoon, so far so good.
Hey everyone,
I updated the script (using version 0.6.1-ben), and it seems to be working for me now…you can find the new code at the following location:
http://ben.pastebin.com/gmdibjrj
Please leave a comment here and let us know if it does or does not work for you.
I installed 0.62 version 2 days ago and it seems to work fine. See it yourself
http://www.sdtv.gr/dreamMemcron/index.php some hours
http://www.sdtv.gr/dreamMemcron/index2.php one day
http://www.sdtv.gr/dreamMemcron/index3.php three days
Thank you for your effort
Hello,
I’m just curious as to whether this software will be updated anytime soon? Hope your not drowned by stupid, unnecessary feature requests and hunting bugs.
Cheers,
Andreas
First off, thanks for developing and maintaining memcron!
I just noticed that the VPS > Manage Resources page of my Dreamhost control panel now includes a CAPTCHA type field for changing memory allocation of my personal server. Here’s a screenshot: http://esyntropy.com/memcron/dh_resources_captcha.jpg
Anyone else notice this? I’m wondering if this will inhibit memcron to adjust memory allocation automatically and “behind the scenes.”
Looks like they’ve taken that out – at least it’s not in my account…do you still have CAPTCHA in your account?
@Chris I just checked and the CAPTCHA field is still in my account. For what it’s worth, it doesn’t seem to inhibit memcron. It’s been working great for me
I was using .6 then I tried .6.1-ben and I keep getting the same issue: Warning: fopen(/home/[username]/[site]/memcron/logs/psxxxxx.csv) [function.fopen]: failed to open stream: No such file or directory in /home/[username]/[site]/memcron/index.php on line 94
Can’t open log file!
I check the logs directory and there is a psxxxxx.log file (empty) but no psxxxxx.csv file.
Hi,
I’ve been using memcron for a few days now, and i’m graph is fine, but my RAM isn’t adjusted.
As anyone any idea of the problem ?
http://www.deadasdisco.org/mem/
Thanks
From your graph, it looks like it’s working now – is this true, or are you manually changing the resources?
Thanks for your answer !
Actually, i liked Memcron but finally installed PSManager which worked instantly :/
I keep Memcron to see the graph … If you have any idea …
I’m not very familiar with swap, cache, free memory, and don’t understand why even if my memory never exceed 400Mb, PSManager or Memcron still allocate about 800Mb…
You may just need to play around with the settings in the config.cfg file…try these:
$mem_free_confidence = 0.75;
$mem_free_to_used_ratio = 0.7;
Hi,
I have tried everthing to get Memcron to work on my DreamhostVPS, but so far without luck.
I use these settings:
Cron-command at DH admin:
perl ~karstensommer/sommerconsulting.net/memcron/memcron.pl
Should be accessed at:
http://www.sommerconsulting.net/memcron/
Log is writeable (755) and the user has Shell/FTP-access, so it should work right?
I hope that someone would be kind to help – I badly need control of my memory!…
/Karsten
Hi Karsten – I checked out your memCron link and it looks like it’s working. Did you get your issues resolved?
Yes and No
I looks like it is working, but when I put my email in the cron-job to send me reports, i get this:
main::normsinv() called too early to check prototype at /home/karstensommer/sommerconsulting.net/memcron/memcron.pl line 94.
main::normsinv() called too early to check prototype at /home/karstensommer/sommerconsulting.net/memcron/memcron.pl line 95.
*** Warning! Threshold exceeded! ***
Total: 500 Used: 320 Free: 180
Mean: 314 Stdev: 65.40 Load: 0.30
Tolerance: 124 Tolerance (free): 636
Target: 956
Memory was resized from 500 MB to 956 MB!
Error calling set_size:
unique_id_already_used
No need to adjust memory!
—–
So there is still an error somewhere…
Which version are you using? I’m using “0.6.2-chris” (http://ben.pastebin.com/gmdibjrj) and it seems to be working perfectly fine.
I guess I’m using an older version then – 0,6. I just downloaded the one I found at this page. I’m very new in SSH /Cronjob, so perhaps I did something wrong somewhere…?
The reason that I wanted to use memcron was because it was recommended to me by a Dremhost Supporter due to heavy memory use… So I really hope that I can get it to function properly
Give version “0.6.2-chris” a try. I think it’ll solve all of your problems.
http://ben.pastebin.com/gmdibjrj
Now it’s seem but work!
Thanks..
I’m just a bit concerned about that it is setting the memory way to high:
Total: 809 Used: 341 Free: 468
Mean: 249 Stdev: 85.60 Load: 0.04
Tolerance: 162 Tolerance (free): 627
Target: 968
I’v set it to 500 Mb at dreamhost, but it looks lit set much higher. I’m not sure what to look at in http://sommerconsulting.net/memcron, but I really can’t belive that i’m using more then 200-400 mb…?
—-
You probably need to adjust the two settings:
$mem_free_confidence = 0.650;
$mem_free_to_used_ratio = 1.90;
The lower those two numbers, the less free memory is reserved. If you’re not getting big jumps all at once, you might try low numbers like 0.25 and 1.20. Just realize the lower they are, the slower memcron will be to adjust your memory upward and the less stable you site may be if there is sudden demand.
As I understand it, a mem_free_to_used_ratio of 1.9 will try to have almost twice as much free memory as is being used (so if you are using 400MB, it will want a total of 1160MB) – I have this value set to 0.7 on my server…like Bret said, I would just play around with those two settings until you find a configuration that works well with your site(s).
Good luck!
Chris – are you sure you have set it at 0,7. I just tried, and after I while I got a mail from the server, that it was not possible to downsize memory to 4xx, when it used 5xx?
Someone please correct me if I’m wrong, since I haven’t had the time to confirm this…but I think this is because memcron does not take into account any buffer & cache memory, which the dreamhost API does…so, for example, if your resources are set to 500MB, memcron may think that your server is only using 200MB (which is true), but dreamhost is taking into account the other “used” memory and thinks you are using almost all of the 500MB, and will refuse to downgrade the resources…I think memcron will need to be changed in order to force a resize – if this is available in the API (although a forced resize, according to the dreamhost panel, may cause a server reboot)…I think the reason I hadn’t encountered this error until now is because my server has stayed constand at 300MB.
Can anyone else confirm this or offer any more information?
PS: you can SSH into your server and run the “free” command to see the true amount of memory you are using (on the second line of output).
Thanks – that should be it!
I had $mem_free_confidence = 0.99;
$mem_free_to_used_ratio = 2.00;
Thanks both of you…
James, maybe you guys could just upload memCron to GitHub and maintain it from there? Wouldn’t that make things much easier for you? It’s open source anyway, isn’t it?
Hi, I just tried this utility and receive this error / notifications:
main::normsinv() called too early to check prototype at /home/../../../memcron/memcron.pl line 94
And also:
Error calling set_size:
unique_id_already_used
No need to adjust memory!
What is that? Is that normal? Or is there something wrong?
I work for Dreamhost and this script causes way more issues than anything else. I am constantly removing it for people. Don’t use this shit.
So maybe Dreamhost should provide its own script ! Monitoring and auto-adjusting memory are basics needs and i don’t understand why i can’t doing it directly from my admin.
Here’s the thing. Some applications cause way more ups and downs in memory use than others. If you are going to use this, best practice would say you should turn off updating and use it only to monitor for a while, adjusting the parameters so that it’s auto-adjustments are more along the lines of what you’d do yourself if you were watching it all the time. When it seems like it is doing it right, then and only then should you enable updating.
It is difficult for any software author to guess at what limits are tolerable and how reactive the script should be for you and your server. Calling something worthless because users haven’t taken the time to learn how to use it isn’t helping anyone.
I haven’t used this other than to monitor memory for a while, but have been wanting to get back to it. The issues I’ve seen recently are its speed and it can’t seem to resize the memory down to what it wants. So I’ve taken the 0.6.2-chris version and made some modifications: (1) remove the list_ps code to get memory. This speeds up the process tremendously. (2) replaced the memory routine with the one from psmanager (GPLv3 license) which also seems faster and more reliable. (3) added a new configuration parameter $force_downsize_ratio which defaults to 0.20 meaning that if it will save more than 20% of currently allocated ram, it’s OK to force the downsize potentially causing the VPS to reboot. You can set it to 0 if you want to disable the force feature. Because the speed is better, you can now schedule this more often to make it more accurate and responsive. To do that, you’ll have to edit your crontab manually. You can get version 0.6.3-Bret here: http://pastebin.com/zw6Vbe2v
Hi Bret!
Digging the revisions, but I’m getting a funky error. The cron reports (sensitive info removed):
I also got a fail notice from memcron directly. Bottom line: it appears memcron is identifying when limits should be changed but it’s unable to change it.
Any idea what would be causing this?
Thanks in advance,
–eric
Quick update: It appears that scaling resources UP works with no problem, but scaling resources back down still yields that “Odd number of elements in hash assignment” error…
sigh…suddenly memcron has broken.
any ideas on this?
main::normsinv() called too early to check prototype at /home/user/hinternet/memcron/memcron.pl line 94.
main::normsinv() called too early to check prototype at /home/user/hinternet/memcron/memcron.pl line 95.
oh, I see this is a recurring problem. OK
OK, memcron is actually working, as others have said it is a warning only.
You can stop getting emailed by adding ,code>> /dev/null to the end of the cron line
e.g. perl $HOME/hinternet/memcron/memcron.pl > /dev/null
Hello guys, really nice script, just what I needed.
I will give a try on this one before try the VPS Manager from 7is7.
Well, for now it’s working, it does change memory, etc.
The only thing I have to understand is the Tolerance (Free).
I can’t get that to understand that. Could any one explain? Thanks….
Other thing is, every time my website uses more memory, VPS just goes downtime and even if I change to more memory, let’s say, 4000MB, it won’t go back up unless I reboot! And so the script won’t work because of that.
I would like to know if I can install the script on another site, like my other shared website, and it would be always on and in case would reboot the server if it would be off, like I said.
Thanks.
@Eric: I really can’t explain tolerance. I recently moved back to active management after a long time of just using memcron to monitor the memory. I made a number of changes to it to make that possible. Now it runs every minute, but once a change is requested, it won’t request another for at least 5 minutes. Even at that, it’s not as reactive as I’d like it to be. If anyone is really interested, I can post my latest version again.
There are some issues you should know about. DreamHost runs a process monitor and a VPS freezer. Essentially, if your VPS exceeds the allotted ram, the process monitor will kill the active process(es) to get the memory usage back down. They recently introduced the freezer so that if you repeatedly exceed the allotted ram, your VPS gets frozen for an unstated period of time, seemingly around 30 minutes if my experience is any indication. The bad part of all this is that there is no indication that any of this is done anywhere you can see it, so you don’t know it’s even happening. Because of that, you can’t run your memory close to what you see in the graph or you’ll end up with a frozen VPS. I recently had this trouble with one VPS that runs one drupal-based website. The graphs show about 35% memory usage but they claim that’s because the process monitor kills stuff before it shows up. Anyway these are issues you have to keep in mind. More changes are coming to VPS memory management per DreamHost support, so perhaps some of the issues will be mitigated eventually.
Hey Bret, I think would be very nice of you to post the new version.
And about this freezer, I just hate it.
They say it takes 5 minutes frozen, but there was one day it was down for 3 hours and I had to reboot.
OK… Here’s the link for 0.6.5-Bret which includes the limit of not updating more often than once in 5 minutes. http://pastebin.com/Ub0ERAvR
And in case it helps, here are the settings I use:
$cron_interval = 1;
$max_memory = 1000;
$min_memory = 700;
$mem_threshold = 800;
$alert_on_change = 1;
$alert_on_error = 1;
$mem_free_confidence = 0.50;
$mem_free_to_used_ratio = 0.60;
$downsize_resistance = 30;
$num_samples = 30;
$num_data = 30;
$num_data_csv = 8000;
One more comment on this. I do have a script I use to monitor my VPS machines. I used to have it reboot the machines if they continued to be unresponsive, but I disabled that when DreamHost got rather unreliable for a while. The script is VBScript and runs on a Windows box, so probably not exactly what you’re looking for. The code is still there to do the reboots–it’s just commented out. It’s something I can share ASIS if you’re interested.