memCron 0.4.1 fixes memory issues

memCron has been updated to address the 2x increase of swap memory recently implemented by DreamHost. So far, there hasn’t been any official confirmation of this change. I’m guessing DreamHost is still experimenting with this new change and doesn’t want to make any commitment to support these extra swap memory just yet.

Here are the changes I’ve made in this minor update:

  • Added: favicon for easier recognition of memCron tab in browser
  • Fixed: Memory calculation due to the 2x increase of swap memory by DreamHost
  • Fixed: Proper max axis value for “Used + Total Memory”

Download: memCron 0.4.1 (32.57 KB)

Please give this update a try and let me know if there’s any issue. I have tested this on my own PS server and it seems to work just fine.

220 thoughts on “memCron 0.4.1 fixes memory issues”

    1. Hi Eduardo.
      I did the upgrade in the following way:
      - I deleted the file config.cfg and the logs folder from the new downloaded memcron.
      - Then, I uploaded all other files to my PS, so I didn’t lose anything.

      This works for me.
      Maybe works for you too.

    2. The only files that changed in this release are:
      index.php
      memcron.pl
      favicon.ico

      So if you upload/overwrite those files to your memcron DIR you should be all set.

    3. Alex’s method works, but since the logs folder that comes with memCron is empty (except .htaccess) you shouldn’t need to remove the folder before uploading.

      Deleting the config.cfg works for this release, but if future updates introduce some new features then you should manually port either the new config values to your old config.cfg or vice versa.

    4. Here‘s a new version that replaces most of the backtick calls with builtins — I’m calling it 0.5 but Yaosan can do whatever he wishes with it. I’ve also swapped the Statistics package with a stripped down function that does the same work.

  1. Hi. This plugin is awesome!
    I would like to suggest a new feature.
    As you problaby like to receive doantions, it would be a nice feature if memcron could calculate how much money were saved. This would make more clear the real value of this plugin.
    Just put a new option on config file : ” $default_without_memcron = 400; ”
    Then, use the diference between this value and the value adjusted by memcron to calculate the money saving.
    Is it a good ideia?

    1. Hi Alex, I agree that it will be an interesting number to know if I do implement a feature like that. Definitely something fun to add to memCron when I have the time to do it, and if you’re right, it could be helpful to encourage more people to donate the money saved! :P

  2. Thanks! Works perfectly! My donation will be, next month, the difference between what I use to pay to DH, and what I start to pay before memCron. Fair enough isn’t it? ;)

  3. Ok, so I have been running v0.4.1 since last night, and I have noticed that my memory has not moved from 150MB even thou my total used memory has exceeded that number. See screenshot: http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-041.png

    As you can see from the screen shot my memory exceeded 150MB at the following times:
    6:20 – 153MB
    6:26 – 153MB
    7:12 – 151MB

    Not by much, but still, my config.cfg file should be setup to increase this when the memory gets close to the 150MB number. My config.cfg file is setup like so:

    $cron_interval = 2;
    $max_memory = 512;
    $min_memory = 150;
    $mem_free_confidence = 0.999;
    $mem_free_to_used_ratio = 1.6;
    $downsize_resistance = 15;
    $num_data = 60;
    (only relevant variables posted)

    So is memCron now considering that SWAP space as extra memory, do I need to set my $mem_free_to_used_ratio above 2 in order to have memCron adjust and not use SWAP space?
    I am guessing this is the case, but I just wanted to check in here first.

    1. Hi BassKozz, you shouldn’t need to change any of your config value for this new release, if these settings work for you before they should work just the same now.

      The only difference is the extra swap memory that DreamHost provides now. What it means is sites that are not using memCron will now be less prone to downtime when there is a spike in usage. As for memCron users, the extra swap space is used to reduce the margin required for setting the total memory because of all the extra free memory that is now available.

      It doesn’t really matter if swap space is used throughout the day and you shouldn’t worry about increasing $mem_free_to_used_ratio just to prevent that from happening. In fact, the swap space is given to handle the spikes so when memory do go over your requested total memory, swap memory will take care of the extra load.

      The bottom line is only increase your $mem_free_to_used_ratio if your site experiences huge spikes on daily basis. The higher you set the ratio, the better memCron can handle those spikes.

      1. But the problem is it’s not acting as it did before…
        Before memCron would adjust my memory if the total used came close my actual memory (not including swap space), now it’s not making that adjustment. My assumption is because of the NEW expanded amount of SWAP space. But as you can see from my graph (http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-041.png) the adjustment isn’t being made when it hits that threshold.

        I guess I am going to have to do some more testing and see if I can get my PS to spike in order to force memCron to resize, and make sure it’s working properly.

        1. Hi BassKozz, The difference you’re seeing is just a result of the extra swap memory, fundamentally nothing is changed. The reason why memCron doesn’t change your total memory when used memory gets close to it is because no changes is necessary to satisfy both the $mem_free_confidence and $mem_free_to_used_ratio you set. You have so much extra swap memory now that no changes is necessary unless the used memory grow significantly above your total memory requested.

          Take a look at the max value on the used + free memory axis (left axis by default) to have a feel how high the used memory needs to jump to before your server will run out of memory. For 150MB total memory requested, used memory can grow up to 450MB before your server starts giving internal server errors. There’s no reason for memCron to increase your total memory when your used memory is just say, 160MB. At this value, you have 290MB of free memory, which correspond to free to used ratio of 1.81 (way more than enough).

          I hope this clears things up. ;)

          1. Ok, now I get it (sort of)…
            Isn’t swap memory slower then regular memory [RAM] (generally speaking)?
            My understanding of swap memory is that it’s just hard drive space that mimics RAM, and RAM is much faster then hard drives. So isn’t the whole point to avoid using swap as much as possible.

            Now I am not sure how DreamHost’s swap memory is handled but I assume it’s using some form of fixed disk.
            I guess I’ve always been under the impression that memCron helps one AVOID using swap space, therefore allowing the site to be more responsive (i.e. FASTER). But from what I am gathering from your replies is, that is not the case?

            On another (related) note, I think it would be clearer if the Total Memory included the swap space. So the user can see how close there Used Memory is getting to the TRUE Total Memory.

          2. I agree with you that the swap memory is most likely stored in some disk space but I don’t know how much slower they are compared to the real RAM. I’m guessing there shouldn’t be that much of a difference, as long as your site loads within reasonable amount of time.

            memCron is so versatile that it is completely up to the user to decide if they want their site to be super fast or they just want to save money while keeping a fairly high server uptime. If you want your site to be super fast by not utilizing any swap memory under any circumstances, then by all means, increase the free to used ratio to say 3 or higher. For those who doesn’t have a high hosting budget, any ratio higher than 1 should be more than enough.

            The reason why total memory doesn’t include swap space is because this gives a clearer view whether swap memory is used at all. The real total memory is defined by the max y-axis value.

          3. The real total memory is defined by the max y-axis value.

            Ahh I see now… That makes sense.
            I think I am going to set my $mem_free_to_used_ratio=2.75 That way I will only use a little bit of swap (if needed)
            Thanks for the clarification Yaosan :D

      1. When I do perl -MLWP -le “print(LWP->VERSION)” I get:
        5.805

        I created an admin user on my vps and installed libstatistics-descriptive-perl using sudo apt-get. Now I do not get the error from the screenshot, but the script doesn’t seem to do anything at all… I enter: perl memcron.pl and it just sits there with no output.

        1. It’s not supposed to give any output. Did you read the instructions: http://memcron.com/install/
          Specifically the part that explains setting up a CronJob?

          To check and make sure memCron is working check the /logs/ directory within your memCron install directory to make sure it has created a file names psxxxxx.log (where the ‘x’ is the number of your PS)

          If it has then give it an hour or so to populate enough data to create a graph (and create a psxxxxx.csv file in the /logs/ directory) and then you should begin to see a graph when you visit your memCron site.

          1. Looks right to me, but I can’t tell for sure from that screenshot…
            Have you made sure to select every 5 minute interval:
            0,5,10,15,20,25,30,35,40,45,50,55

            The best way to find out if it’s working is to put in your email address in the Email output to: field and if you get an email every 5 minutes then you’ll know for sure it’s working.

            Remember you will have to give it an hour or so to collect enough data before you start to see any graphs.

      1. >Have you made sure to select every 5 minute interval

        I think that might have been my problem! Haha, I only had it set to run on the 5th minute of every hour. That’s why it only collected one data point so far. I’ve got it set properly now, so I’ll check again in about an hour to see how it’s going.

  4. Any ideia on this problem?
    http://i29.tinypic.com/2lwp450.jpg

    From 11:00 to 12:00 I’ve raised the memory manually. After that, i’ts automatic. Look that my memory usage is higher that the blue line and memcron isn’t adjusting.

    There is any option I need to change? $mem_free_to_used_ratio, maybe?

    1. People often get confused, the blue line is just the total memory “requested”. You still have swap memory to handle spikes that grows above that threshold. And since the swap memory is increased 2x, used memory are allowed to grow over that threshold even more while keeping a high free to used memory ratio.

    1. Not sure why it happens, I’m guessing you’re running out of daily resize limit. memCron tries to reduce your total memory but it keeps failing to do that. As you can see, your memory is stuck at 1041MB even though memCron tries to reduce it to around 400MB based on your actual memory usage.

      Try increasing your downsize count to a higher value if you’re running out of daily resize limit.

        1. what kind of traffic are you getting? my DH ps is sticking around 200-400MB usage and my sites that are running on the VPS get about 5,000 unique users a day. Are you using apache or lighttpd to serve the static content?

          1. He’s using Apache.

            All of my sites were using 1000-4000MB on Apache. On top of that, the larger the files, and greater the amount of downloads on them at a time, greatly increases the mem usage, which is why I was hitting 4000MB during peak hours. Under lighttpd with large downloads it would hit 1000MB peak, 300MB avg. After cutting all large downloads from my sites (no more 20-100+MB files) my mem usage is down to about 100-120MB avg.

            Apache with no websites online uses 500MB.

            You could probably do quite a few tweaks to get your usage down to my level, choob. I’m hosting all of my services with lighttpd, not just static content. That includes the wiki, wordpress and boards for CS, MG and TEXTCHAN.

            If you host any email on your server then you should consider switching to the Google mail option.

          2. So John,
            What did you do with all the large download? Stop providing them, or move them somewhere else? Just curious. Apache on DreamHost seems really stable when it comes to large downloads, and it isn’t pushing my memory up terribly high. I tried moving them to lighttpd and found that every download seems to get read into memory before it’s sent, so it only takes a few concurrent requests to kill lighttpd completely. I suspect DreamHost needs to makes some adjustments to its lighttpd config and I’ve got a months-outstanding support request for it, but it’s obviously not high on their priority list.

          3. Simply, I moved them to some shared hosting.

            The stability on lighttpd is great and you don’t get lagged anymore by any retard they put on your server.

            The shared hosting is just dandy for your offsite fileserving needs.

  5. Hi,

    Thanks for this! I am continuing to get a error on line 91 that it can’t find the CSV file. I am assuming this file will be automatically generated, but after a few hours I am still getting the error. Any help is appreciated.

    Thanks!
    -p

    1. Ignore my comment, it helps to have the ‘perl’ command before entering the path :)

      memCron is amazing! Thanks again for this!

      -pg

  6. I’ve been testing the memcron since this fix and I can take one conclusion: When my PS hit the Swap memory, I have high loads (sometimes 5.0 of load) and then the website starts to get slow.

    This is really bad for me, because memCron did not raise my memory when it’s needed. But I’ll change the config files to fix this for my case.

    Just commenting this here for anyone that is having the same problem (or not).

    1. Hi Fred, do keep in mind that memCron merely acts upon what you configured it to do, as such you shouldn’t claim it didn’t raise your memory when it’s needed. As you have pointed out yourself, changing the config value and increase the free to used ratio will easily solve this issue for you. ;)

      Personally I haven’t notice any problem with swap memory causing high CPU load. The memory is there for you to use, and if you somehow come to the conclusion that using swap memory is really BAD, then you can simply increase the free to used ratio and pays a premium for that extra buffer.

      1. It’s really a config problem, I’ll change it and everything will work. Good to hear that Swap isn’t causing problems for you, I’ll check my server configuration, change some scripts and test them to see what heappens.

        Anyway, thanks for making this excelent memory manager!! memCron is amazing!²

        For me, DH should make this default to the all PS servers! :)

  7. Now I’m seeing a very strange phenomenon with memcron. Every time it adjusts the memory up, it adjusts it way above what it needs, then 5 minutes later, drops it back down again. You can see this at http://www.wcgweb.org/memcron-bounce.jpg.

    Here are the settings:

    $mem_free_confidence = 0.650;
    $mem_free_to_used_ratio = 1.90;
    $downsize_resistance = 12;
    $num_data = 80;

    Essentially, DreamHost has asked to try to run with a little more real memory than is used. Actually, they blamed the fact that I was getting no response from the server on memory when I still had 800+MB swap space left. Anyway, I’m trying to get memcron set so it tries to keep real memory just above used memory. Not trivial in 0.4.1, it took a lot of experimentation with the settings.

    Now it works, but I’ve got the “bounce” every time the memory needs to go up. Any suggestions to eliminate that?

    What would really be helpful to me is rather than ratios that are a bit hard to understand and tweak, we had a setting like $real_memory_available_target=10; where memcron would try to keep around 10MB real memory free, and $real_memory_available_min=3; where it would adjust it up any time it fell below 3MB. That would make it easier for me to use. I suspect that some others might find that helpful as well.

    1. To answer my own question here, it appears the “jump” I’m seeing in the graph is a matter of how memcron LOGS the memory, not the actual memory allocation. To that end, it is a bug and should be reviewed and fixed in time so the graph reflects reality, not the skewed calculation that is currently being recorded.

      1. Yes Bret, it is a bug. I’ll release a patch for it shortly. Don’t worry too much about it at the moment though, it is not skewing any calculation at all, just visual annoyance. :P The “skewed” data is not used for memory calculations.

    2. Hi Bret, the ratio parameter is used because it allows a generic setting for sites of any scale. If I use a hard number for memory allocation, that will be good only for site of certain size.

      Instead of working with hard numbers, I think it is more intuitive to see things in proportion. If you want to calculate ratio based on hard numbers, here’s how you can do it:

      Say you want to reserve 10MB of real (non-swap) memory and your average usage is 300MB, then:

      Target memory = 300MB + 10MB = 310MB
      Total memory = 310MB * 3 = 930MB
      Free memory = 930MB – 300MB = 630MB
      Free to used ratio = 630MB / 300MB = 2.1

      Take note that calculation of total memory is not always multiply by 3, the calculation of total memory changes when your used memory is larger than 450MB.

      e.g. If your average used memory is above 450MB, say 480MB, and you want to reserve 50MB of real memory, then:

      Target memory = 480MB + 50MB = 530MB
      Total memory = Target memory + (450MB * 2) = 1430MB
      Free memory = 1430MB – 480MB = 950MB
      Free to used ratio = 950MB / 480MB = 1.98

  8. Yason,

    I have tried everything to make Memcron work at my website, but he tries to change the memory, as I am emailed, but It simply doesn’t at the panel on dreamhost PS Resources.

    My config
    $mem_free_confidence = 0.500;
    $mem_free_to_used_ratio = 1.10;
    $downsize_resistance = 8;
    $num_data = 40;
    Others default (CSV 2000)

    http://www.vejaisso.com/memcron/

    My website is really regular in memory use, making Memcron extremely useful, specially at night. Yet, it simply doens’t change my PS usage.

    Thanks in advance for the help, and congratulation on the tool!

  9. Yaosan, may I suggest to amend get_mem_total like so to cater for the fact that memory targets above 450 MiB are topped by 900 MiB swap:


    sub get_mem_total {
    my $mem_target = shift;
    return ($mem_target < 450) ? $mem_target * 3 : $mem_target + 900;
    }

    1. Hi Robert, thanks for the suggestion, I’ve already fixed this issue after some users reported the “jump” they’re seeing right after each memory size change. It will be included in the next update.

  10. Just got Dreamhost’s August newsletter:
    3. New! Major PS graph improvements!

    and the graphs you see there about memory/cpu usage now update about every five minutes.. AND you can even zoom in on the daily graph to see all that data!

    Looks like dreamhost is biteing off memcron :-P

  11. This script is great but i think you should improve it workings a lot for high traffic sites…
    I have a big website hosted that idles at 100MB during night but can peak to 100hits/s using 1800Mb in a minute at 9 o clock every day.

    Ive tried to just edit config with no success on fast responding to this peaks so i alter the logic a bit.

    You should improve this on next revision:
    - Urgent: Separate memory usage from swap usage and get two grapics from that. Show Total as a sum of that and dont save total in csv. Save the mem and swap separatly.
    - Allow config option to include swap on not in calculations.
    - Show swap in graphic.
    - Show calculated target on graph even if no adjustment is made due to down resistance. Usefull for debug!
    - Add javascript zoom to graphic like in DH panel.
    - Include my next logic so it is more responsive to rapid changes demands.
    - Correct this bug: Wrong values for total on graph when target is committed. When memory is changed up i got a spike and wrong values when changes to low.

    My formula to answer fast memory demand:

    my $mem_mean = round( $stat->mean() );
    my $mem_stdev = round( $stat->standard_deviation() );
    my $mem_max = round( $stat->max() );

    my $mem_used_tolerance = round($mem_used_zscore * $mem_stdev);
    my $mem_free_tolerance = round($mem_free_zscore * $mem_stdev);

    unless ( $mem_free_tolerance > ($mem_mean*0.2 + $mem_used*0.4 + $mem_max*0.4) * $mem_free_to_used_ratio ) {
    $mem_free_tolerance = round( ($mem_mean*0.2 + $mem_used*0.4 + $mem_max*0.4) * $mem_free_to_used_ratio );
    }

    print “Mean: $mem_mean\t Stdev: $mem_stdev\t Max: $mem_max\t Load: $cpu_load\n”;

    Mean: $mem_mean Stdev: $mem_stdev Max: $mem_max Load: $cpu_load

    My config:
    $cron_interval = 2; #i run it every 2 minutes
    $max_memory = 4000;
    $min_memory = 150;
    $mem_threshold = 500;
    $mem_free_confidence = 0.9;
    $mem_free_to_used_ratio = 1.2;
    $downsize_resistance = 10;
    $num_data = 15;
    $num_data_csv = 44640;

    1. Hi Chaveiro, interesting suggestion you have there, particularly the change of the calculation algorithm. I’ll take a look at that and see if it does work better. Since I don’t have an extremely popular site to test the settings, I’ll need more feedback from heavy users like yourself.

      FYI, the solution I proposed in the past is to set up a separate cron job to change the min memory memCron can request if you know exactly when the spike will hit your server.

      I still don’t really get the whole argument of not including swap memory into the calculation algorithm, but if the separation of swap and guaranteed memory makes some people happy, I’ll consider adding an option to not include swap in memory consideration in upcoming versions.

      The target memory will definitely be added as one of the “plotable” data in upcoming versions, if not the next update.

  12. btw, for those who doubts memCron’s ability to handle usage spike using the current algorithm, do take a look the demo right now: memCron demo

    And here’s a screencap of it for future reference: memCron reacting to memory usage spike

    To be fair, some of the smaller spikes that happened before the huge one did help memCron “prepare” for that huge spike. Overall, unless it’s a split second spike, memCron should work quite well if you leave enough margin for it (and the margin doesn’t even have to be big, around 1.2 free to used ratio should be sufficient).

  13. Yaosan, the memCron works great for most of the sites with gradual changes.

    I’d does not work if you dont have resources to run it though.

    My users are kind of erratic in the morning.. i get spikes from idle to huge accesses rate in a split second, like i said before, this poses a big problem if all the swap is used in that spike… (from 150MB idle to spend all the swap i only needed 10hits/s without changing memcron) and it did occour to me in several ocasions.
    The server got so full with no resources that not even cron / memCron / free was able to run anymore..

    Apache threads didn’t have free memory to start and the server bogged until hits stop.
    As hits dont stop here, the server will be all long in this situation with high cpu usage and denying requests.
    I had to manualy change ram from pannel one time worked other only with a reboot.

    My formula is not a miracle, if you happen to expend all your swap and cant get memcron to change ram and hits dont gives server time to recover and free any swap, you’re ‘crashed’!

    That’s why is so important to have more control on the swap separatly from the real mem, specialy when idling at 150MB as swap space is very low then and a litle spike can use it all and put you in this condition.

    When you have much memory alocated and expend it, as swap is also more, server is very tolerant to spikes.

    Another isue about beeing on the limit with free real memory close to zero is with apache …

    Apache starts more threads on demand and sets an initial maxclient value based on free ram from time to time. And it restarts alot!

    If free ram is kept low it is more conservative only allowing you to serve as many requests as your server ram can handle (not knowing you can give him more ram on demand) and in some ocasions starts denying requests due to limit max clients connected (not because not having free swap).

    So your server ram and swap usage does not increase much because your’e rejecting hits. This condition is dificult to detect from graph as you only se a gradual increase in usage like any normal utilisation but your users can be being regected!

    This is easy to see in apache error.log as anyting like this:

    [error] server reached MaxClients setting, consider raising the MaxClients setting
    WARNING: MaxClients of 215 exceeds ServerLimit value of 93 servers,
    lowering MaxClients to 93. To increase, please see the ServerLimit
    directive.
    [notice] Graceful restart requested, doing restart

    About the isue of not havinf free swap you will see things like this:
    [notice] mod_fcgid: call /home/…memcron/index.php with wrapper /dh/cgi-sys$
    Out of memory (Needed 8160 bytes)
    php (pre-forking): Cannot allocate memory

    Regards!

    1. I am interested in your problems Chaveiro,
      While I don’t have any sites that experience the amount of traffic (spikes) that your seeing I hope that one day I will (sites grow in popularity), and I am curious, what if memcron wasn’t a cronjob. Meaning what if memcron stayed active in resident memory and ran 24/7/365 (non-stop) to check the status of available memory and adjust accordingly. Rather then have to wait to run ever 2minutes. Would that resolve the problem?

      Just curious to hear your thoughts.
      -BassKozz

      1. Maybe…

        Having it running non stop is ok if you can ensure to restart if it exits for any reason. Server restart / crash…

        The only problem i can see is that it runs free command regulary and in this sitiation if it cant alocate memory to run free it will crash and exit…

        I personaly think there is no benefict on making this perl application a deamon as if it depends on other commands.

        Best way would be to create a deamon in C that loaded all required functions to ram and dont depend on external commands. So it will always have resources alocated even if no more swap is available.

        Other alternatives… Run memcron from other machine (shared hosting) and inquire PS server status via http to a simple api that returns the resulf from the free on the PS server.

        Regards,
        Nuno

        1. Nuno is right about the issues with memCron running as a perl daemon. All the other commands required to query the server status won’t work if memory is full.

          I’m not familiar with creating a daemon in C so that could be something for me to explore in the future. For now, your best bet is never let your server running out of memory in the first place, which memCron is trying its best to achieve. :P

  14. Sathurday Oct. 17 memcron stopped updating my PS. I discovered that a CAPTCHA was added to the PS resource management page, so access via API did not work.
    I contacted support for an explanation.

        1. This is what I wrote to support:

          > 1) I use a tool called memcron http://memcron.com which uses the DH API
          > > to set the PS memory. I scheduled it to run as a cron job every 5
          > > minutes (the default).

          And this is the reply from DH support:

          Ah, OK. Since Memcron uses the Dreamhost API, it should work fine — the
          CAPTCHA is only applied to changes made from the panel. Running Memcron
          every five minutes is probably overkill, though, as you are limited to 30
          resizes per day
          .

          The official memcron documentation sets a default 5 minute interval in the cron job. Is that a new DH policy? Has anybody had issues?

  15. Hi Jan, it doesn’t matter if you use API or any other method to change the PS memory settings, as soon as you are making more than a couple changes a day, the CAPTCHA will be added to the panel.

    Back the to overkill comment by the DreamHost support, running memCron every 5 minutes is probably more of an “underkill” than overkill. We have power users running memCron every 2 minutes and they didn’t run into any issue with that. memCron is capable of changing the criteria of adjusting your memory based on how often it is being run, as long as the $cron_interval is set properly according to your cron job settings.

    In short, there’s no need to change the default 5 minutes cron interval suggested.

  16. Memcron seems to want to adjust memory a bit more often than I’d like too. There is a setting you can use to make it a little more resistant to change:

    $downsize_resistance = 12;

    Basically, it takes 12 consecutive runs where it wants to lower memory before it does. At every 5 minutes, that means it takes an hour of wanting a lower memory setting before you get one. If that’s too often, you could adjust that higher– like 18 for 90 minutes or 24 for 2 hours. Any higher than that and I’d expect you’re not getting the savings you want to see from memcron.

    I’m using a different manager on one of my three PS machines because memcron wasn’t quite doing what I wanted. PSManager wants to run every minute, but it also has thresholds that must be exceeded before it will raise or lower memory to keep within the 30x per day limit. On that one machine, I have large random spikes in memory usage and PSManager just seems to make it more stable. The other two don’t have that issue and Memcron does exactly what I need.

    1. Interesting case you have there Bret, ideally I would like memCron to be the ultimate memory management solution i.e. it should work with any kind of servers, whether they have large random spikes or not. All the user need to do to accommodate different kind of server is with a little tweak in the config values. Apparently, memCron isn’t quite good enough to get there yet.

      What exactly happens when you run memCron on that server with random spikes? What is stopping memCron from being more stable than PSManager, is it the frequent downsize or memory isn’t rising fast enough to handle the spikes?

      Let me know if you have any suggestions.

  17. The CAPTCHA has been removed from my panel. I asket about the limiti of 30 resizes, this is the reply:

    This limitation is now documented:

    http://wiki.dreamhost.com/API#dreamhost_ps-set_size

    It was previously alluded to by the “exceeded_30_resizes_today” error,
    but wasn’t explicitly mentioned in our documentation.

    I agree that a default memcron config should not that many resizes, and mine was a very conservative config with default downsize resistance 12 (I see memcron as a way to ensure that the PS has MORE memoruy, rather than a tool to save money).

  18. I think it just doesn’t see the spikes because of the 5-minute interval. I’ve uploaded my current memcron graph (http://wcgweb.org/memcron.jpg), which has been modified to show the target setting for memcron while it monitors the memory allocated by psmanager. Then there’s my log from psmanager (http://wcgweb.org/psmanager.log) which clearly shows the spikes and the reasons for the increases in memory that you can’t see on the memcron graph. Because it’s all lined up so nicely, the columns could use more explanation: timestamp, total memory, memory used, memory free, total swap, swap used, swap free, new size = what psmanager has set the total memory to, optimum size = what psmanager thinks the memory should be. I’ve toyed with the idea of running memcron every minute to see if they match up better, but haven’t actually tried it yet. I’m not super happy about the high memory usage at the moment, but I can see that it’s just trying to prevent the machine from running out of memory when it needs it.

    1. Interesting mod and observation there, Bret, thanks for sharing that with us. I might consider setting 1 minute interval as memCron’s default in the future and shorten the memory data collection process to 30s, or at least provide some recommended usage settings for aggressive users. To do that, first I’ll need to integrate cron job management into memCron’s code since you cannot set cron job that runs more often than every 5 minutes in DreamHost web panel.

      For power user aiming for maximum uptime, an option to exclude swap memory from memCron’s memory calculation should also provide more room to deal with spikes in usage.

      Hopefully with both of these changes in the upcoming version, memCron will work better for any kind of server regardless of their loads.

    1. I’m not a MySQL PS user myself but I voted for it anyway! ;) I’ll definitely need some people to help me test memCron on these MySQL PS when this gets implemented.

  19. Dear Yoasan,

    Before I get started I would like to give you my many thanks for such a great tool. Ever since I found out about its existence (After endless searching) I could successfully install the program that has fixed the HUGE inconvenience of leaving my site with insufficient memory when different spikes occur on my site. Memcron has been able to give my site the memory it needs when it needs it as you can see in the following screenshot:
    http://i213.photobucket.com/albums/cc277/bloggea2/Screenshot-08_11_20098_40_58.png

    But in the screenshot you can also see the problem that I don’t know why   Memcron takes so long to reduce the extra memory even after 4 hours after the spikes have occurred even after changing the downside resistance to 6.

    What configuration do you recommend for my copy of memcron? (with what you see int he screenshot)

    The only thing i have configured due to defect is the downside resistance to 6 but I have left everything else exactly how it is suppose to be, Since the behavior of my site is “always the same”. Once a day it does receive these spikes that can reach 3000MB (due to visits and robots) then it falls to normal numbers as you can see in the screenshot.

    I would greatly appreciate your help and answer as soon as possible to what I can do so Memcron does not keep unnecessarily my sites memory to high..

    Kind Regards..

  20. Update:
    I am now experimenting an amazing and unexplainable spike in the memory of my server. I have come to have a rate of memory of about 400 to 500mb and one spike a day that can reach up to 3000mb. Now I am having a rate of 2000mb and rising!!…

    WHAT IS HAPPENING?…
    COULD DREAMHOST BE PUNISHING ME FOR INSTALLING MEMCRON?
    HOW CAN IT RISE SO MUCH HAVING THE SAME VISITS AND THE ONLY CHANGE I HAVE DONE IS INSTALL MEMCRON?

    Screenshots:
    http://i213.photobucket.com/albums/cc277/bloggea2/Screenshot-10_11_20099_19_07.png

    http://i213.photobucket.com/albums/cc277/bloggea2/Screenshot-09_11_200917_00_32.png

    PLEASE YAOSAN, I NEED HELP “URGENTLY”..WHAT DO I DO?

    1. Hi Nico,

      I’ll first answer your first question. The reason why memCron allows memory usage to stay high for a period of time before downsizing is because the previous 2 spikes that your server experienced makes memCron more careful about downsizing your server to a lower memory. Technically speaking, the standard deviation increases when you have multiple spikes, which triggers the conservative behavior of memCron as it anticipates yet another spike of similar magnitude.

      To get around this issue, I suggest you play with $num_data to reduce the duration of such effect (smaller $num_data –> shorter duration of conservative behavior).

      And I don’t think DreamHost is punishing you for installing memCron, otherwise they wouldn’t have made it one of the winner of the API contest. I suggest you reboot your server and see if the memory usage improves. You can also run “top” in your shell to see which process is eating up most of the memory. Good luck!

      1. Hi Yaosan,

        Thank you VERY MUCH for your response, it has helped to put everything back in order and now everything is working fine, I re-started the server like you said and everything is now working the way it should. I will keep controlling everything in the next couple of days to see if it goes up again to try and investigate what is causing the problem.
        I am eagerly waiting for the next Memcon version, and i hope by months end to be able to give my grain of rice to contribute what I have earned thanks to Memcron by giving a donation.

        Thanks again!!

  21. I’m woking on it from 2 horus, but I continue to get this error:
    Warning: fopen(/home/USERNAME/MYDOMAINFOLDER/memcron/logs/MYPSNAME.csv) [function.fopen]: failed to open stream: No such file or directory in /home/USERNAME/MYDOMAINFOLDER/memcron/index.php on line 91
    Can’t open log file!
    my cron line for every minute is :
    perl ~MYUSERNAME/MYDOMAIN/memcron/memcron.pl
    and log folder is in 775 permissions

  22. This is the optimal setting I found after some time:

    $min_memory = 150;
    $mem_free_confidence = 0.4;
    $mem_free_to_used_ratio = 2.5;

    It works weell reacting to memory spikes and keeping the pserver always responsive.

  23. Here’s a new version that replaces most of the backtick calls with builtins — I’m calling it 0.5 but Yaosan can do whatever he wishes with it. I’ve also swapped the Statistics package with a stripped down function that does the same work.

  24. I can’t find anything on the Web site about this: in the 12th email underneath the “Memory was resized from xxx MB to xxx MB!” I get the following message:

    *** Warning! Threshold exceeded! ***
    TERM environment variable not set.
    connect to localhost failed (Connection refused) no (more) retries!Uncaught exception from user code:
    connect to localhost failed (Connection refused)
    connect to localhost failed
    connect to localhost failed (Connection refused) no (more) retries!

    1. Sounds like something is screwy with your crontab line, can you please post what you have EXACTLY in your crontab ?
      Or are you using Dreamhost’s Panel>Goodies>Cron Jobs ?
      If so then post the line you have in “Command to run”

  25. For some reason, my level reverts back to whatever setting I have in the Dreamhost PS control panel. Every 12th reading, it will drop down and then immediately jump back up to the level that is set in the control panel. It will never increase higher than my control panel setting.

    I currently am using the default config settings. Any ideas where I am going wrong?

    http://panicstream.com/memcron/

      1. I tried using the panel but am still getting the same error. Note that the memory is NOT being resized by memcron.pl

        Error:

        TERM environment variable not set.
        connect to localhost failed (Connection refused) no (more) retries!Uncaught exception from user code:
        connect to localhost failed (Connection refused)
        connect to localhost failed
        connect to localhost failed (Connection refused) no (more) retries!

  26. Ok, I found that if I add the line “TERM=xterm” to the beginning of my crontab, the “TERM environment variable not set” goes away. However, I still get the following error (and PS memory is NOT being resized):

    *** Warning! Threshold exceeded! ***
    connect to localhost failed (Connection refused) no (more) retries!Uncaught exception from user code:
    connect to localhost failed (Connection refused)
    connect to localhost failed
    connect to localhost failed (Connection refused) no (more) retries!

  27. Love this script. Is there any way to adjust the time in the graph based on timezone? Always confusing to see the graph and try to do the math to figure out when stuff happened.

    Thanks!

    1. Unfortunately, not in the config file. There is a $gmt_offset variable in index.php that you can edit. However, it’s an offset, not a time zone so it doesn’t automatically adjust for daylight time when that changes.

  28. Once again, DreamHost have changed their PS setup.

    My PS started to report all assingned memory ressources as core memory a few hours ago, with zero swap space.

    Apparently, the memory reported by “free” is calculated by either tripling the assigned memory ressources (below 450 MB), or adding 900 MB to the assigned memory ressources (above 450 MB)

    Here a sample output from free with memory set to 458 MB:


    $ free -m
    total used free shared buffers cached
    Mem: 1358 992 365 0 0 0
    -/+ buffers/cache: 992 365
    Swap: 0 0 0

    Same, same, but different with a 342 MB memory limit:


    $ free -m
    total used free shared buffers cached
    Mem: 1026 288 737 0 0 0
    -/+ buffers/cache: 288 737
    Swap: 0 0 0

    Cheers,

    Robert

    1. Try using my modified version of memcron.pl. It doesn’t rely on parsing the output of “free”, instead it reads /proc/meminfo directly so there is no need for those extra calculations.

  29. I don’t think the problem lies within the way memcron determines memory usage but the way the vserver reports core and swap memory. And this is what has changed.

    Here’s a snippet what I get from /proc/meminfo:


    $ cat /proc/meminfo
    MemTotal: 1050624 kB
    MemFree: 684480 kB
    [...]
    SwapTotal: 0 kB

    Basically, get_mem_target and get_mem_total do make false assumptions now and have to be adapted to the new situation, I think.

    1. Gotcha. For what it’s worth, my ps still shows swap space as it always has:
      $ free -o
      total used free shared buffers cached
      Mem: 153600 153600 0 0 0 0
      Swap: 307200 234912 72288

      So I think the target formula just needs to be changed to reflect how DreamHost actually calculates it on the panel page

      burst = mem_total > 450 ? mem_total + 450 : 2 * mem_total;

      Where burst is the amount your memory is burstable by not to.

      1. Nevermind – in 0.4.1 the calculations are correct regardless since it derives the value of swap in get_mem_size() rather than reading it directly. It’s my version that would cause you trouble :-)

        1. Yes, I added that calculation back in when I “upgraded” to your version because by then, my PS machine was already acting this way. One more thing that is causing me great grief is that the values reported for memory actually get corrupted at some point, reporting far less than what is actually there and causing memCron (and psmanager for that matter) to really screw up the allocation. I’d had to disable memory managers until/unless this corruption actually stabilizes. There’s nothing a memory manager can do when one minute the OS report 1665 for total memory and the next it’s 432 with no changes having been made. :(

          For those on the new configuration, please watch your charts closely.

          1. I just updated the version in the pastebin to calculate mem_size/swap based on the total rather than reported values. And I added some sanity checking just in case /proc/meminfo gives weird numbers.

            What I may end up doing is stop running this from cron altogether and just have it run as a daemon. It uses up 8MB of ram every time it runs, but that could probably be cut down to 4MB (or less) if the resizing/email notification were moved to a separate process that’s only called as needed.

          2. I think the sanity check is still allowing a bit of insanity. Memcron keeps pushing me all the way up to the 4gb maximum because every 30 or so datapulls, it gets a huge negative number. See this sample:

            141
            140
            137
            136
            133
            -299946772057164
            132
            131
            132
            152

            Once it gets the -299946772057164, it immediately bumps up the memory to the maximum. Perhaps the logic in the sanity checker needs a little tweak. By the way, I’m using version 0.5.2 from the pastebin. Thanks!

          3. Wow, that’s crazy.

            The kernel on the host server my ps is running on was recently upgraded to 2.6.32.2-aufs2-vs2.3.0.36.28.6-bertlfix1. So now I have the condition that Robert reported earlier. It’s hard to determine what actual memory usage is based on either “free” or /proc/meminfo. But now you can see when you’re bursting through regular memory into swap because swap changes from 0MB to 153MB.

            I’m working on a new version that collects memory from /proc/[pid]/stat directly and the numbers match what’s reported by the list_usage api call. I’ve set up the collector as a daemon that runs outside of cron – so no more 2,3,4 or 5 minute delay between samples. All for less than 3MB resident.

  30. Hello,

    so I have a VPS from DH, what is the first thing that i have to do ? please could u explain me the steps easily? i am a ew beginner in the world of VPS

    thank u

    1. Hello Akirs,

      I would suggest that you get on your shell access and spot check the memory a few times to compare it to DreamHost’s charts. If you have Windows, putty for windows will give you a basic shell. Point it at your domain and log in as one of your users. Then run free -m and compare it against your last entry in the usage chart in the control panel. Do this a few times.

      I say this because the last changes DreamHost made seems to have made the internal memory reporting unstable to the point where it doesn’t anywhere close reflect what DreamHost sees as your memory allocation. If memory isn’t reported correctly, then you CAN’T use a memory manager like this.

      If you doubt this, check out the last two days at http://wcgweb.org/memcron-20100205.jpg. DreamHost during that period reports my memory usage between 250 and 475 while memcron shows wild fluctuations and sticks for long periods around 1600. Note: I have modified memcron so it does not adjust memory–only reports it and what adjustments it would make.

      I do hope that this will not forever be the case. But for now, be very, very careful.

      1. I’m glad I noticed it yesterday and managed to set memcron to min and max 150MB after setting it to that in the DH panel. With lighttpd I’ve been around 100MB or less for 3 months.

  31. I’m having the same issues with DH reporting strange memory figures. On the DH VPS usage page everything looks normal – I’m using about 120-150MB memory. Memcron shows a steady increase of used memory over the last few days and has increased the memory allocated to 650MB.

    So, I rebooted, and dropped the memory to 300MB. Running “free -m” in the shell now shows:
    total: 1502, used: 38, free: 1463
    with swap, shared, buffers and cache all at 0.

    Running “top -c” shows that I’m using a lot more than 38MB, so something’s screwy. I’m contacting DH support.

    1. I contacted DH about this earlier. While the data in their db is correct, there’s a bug in the chart that’s showing inflated numbers. There’s a bug filed and I’m assuming it’ll be fixed soon.

  32. Just so everyone knows, this script can be used on any server to get a pretty memory graph. I had to use Ben’s modified version in order to bypass the Mail::SendMail requirement.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>