Winner of DreamHost API Contest + memCron 0.4!

First of all, I want to thank DreamHost for announcing memCron as one of the two grand prize winner for its first ever API contest (congrats to Dreambot for tying the grand prize!). This is the first time I actually won something for a program I wrote, so thank you for recognizing the quality of the script I developed! :D

The announcement of the API contest results finally brought in many new users who have never heard about memCron up till the announcement. Luckily memCron site survived the spike of visitors without any major issues. I was actually wondering if the announcement will put up a stress test for the memCron script, but in the end the memory usage didn’t change that much at all (probably due to WP Super Cache). Instead, all I saw was a spike in CPU load, jumping the load from 0.2 straight to 13 right after the news was posted on the DreamHost blog. You can still see the spike in this graph generated by memCron. :o

Many of these new visitors were brave enough to give memCron a try and the feedback so far have been mostly positive. Some of the feature requests are pretty insightful and I managed to spent some time to code them into memCron 0.4. After working on it for the past few days, I think it is ready to be released. Please give memCron 0.4 a try and let me know if it works just the way you imagined it would be.

Here is a list of notable changes since version 0.3 (or check out the detailed changelog):

  • Added: New parameter $max_memory and $min_memory to limit the memory range that memCron should operate within
  • Added: New parameter $mem_threshold to set a memory threshold, which when exceeded, will send out notification to $email
  • Added: Base memory and swap memory size to function output
  • Changed: Tooltip for the resources usage graph is now cuter i.e. in yellow! :)
  • Removed: $debug and $username parameter from config.cfg, now DreamHost API only needs API key for authentication
  • Fixed: Memory target are now computed more correctly during downsize
  • Fixed: Total memory is now displayed correctly on the resources graph by accounting for max swap memory, which is capped at 450MB

Download: memCron 0.4 (31.95 KB)

If you find any bug in memCron 0.4, please post them here and I’ll get them verified and fixed asap. If you love how memCron saves you time and money through its innovative memory management system, please consider donating to support this project, thank you! :D

As always, I’m open to new feature request so don’t hesitate to send them in. I’ll try my best to add the most requested feature into the next release! Let’s make memCron the best damn DreamHost PS app in the world! I know it can be done with your support! ;)

144 thoughts on “Winner of DreamHost API Contest + memCron 0.4!”

    1. Hi kwurzel, right now you can go into index.php and change the $gmt_offset to the one that correspond to your timezone, for example the current value of -4 corresponds to EST with daylight saving.

      I will move the $gmt_offset to the config file in the next version. Thanks for your feedback.

  1. I am confused as to why there is a setting in the config.cfg file for “cron_interval”, isn’t this being set via the Dreamhost Panel (Goodies>Cron Jobs) anyways?

    Also I notice that Dreamhost’s panel wouldn’t let me set anything lower then every 5 minutes, if I tried to set it to every minute or 2, or 3, the panel would say “Too many minutes selected!” and not allow the Cron Job to be added.

    I haven’t tested this yet but I wounder if it would be possible to set the cron job via SSH using “crontab -e”, and possibly allow you to set it to run every minute?… Has anyone tried this?

    Thanks for the Great script, and Congrats on the Contest Win :-D

    1. Hi BassKozz, the $cron_interval variable is used in the memory scaling algorithm i.e. changing this has an effect on the calculation results. Setting it in DreamHost panel only tells DreamHost how often to run memCron, but memCron itself needs to know how often it is being run to get things right.

      You are right that DreamHost doesn’t allow cronjob that is less than 5 minutes apart, that is part of the reason why memCron’s default interval is 5 minutes. I think DreamHost discourages running a cron job too often as it may cause stability issues to your server.

      If you do choose to be aggressive, then nobody is stopping you from running memCron every minute using crontab, I haven’t tried it myself but it should be doable. Do take note that memCron takes more than 1 minute to run each time because it will attempt to collect 60 memory samples over 1 minute. This could lead to file locking / corrupt issue when 2 memCron instances overlap.

  2. Thank you very much for you help. That’s totally sufficient for the moment! But anyway, i’m looking forward to 0.5. :-)

  3. Looks good with all the features I wanted.

    While there might be things that need to be added, I would suggest being conservative since we don’t want memcron to be using lots of resources on the server.

    There is a bug with the graph: When I open index.php in Internet Explorer 8, the actual graph lines don’t show, though the axes do. Everything’s ok in Firefox.

    1. Hi Zack, I understand your concern about bloatware, personally I wouldn’t enjoy maintaining a project that is overcomplicated than it needs to be as well. :P Just to reassure you, new features will only be added if many users requested them and only if they make memCron easier and safer to use. ;)

      As for IE8, right now you will need to click on the compatibility view icon beside the address bar to see the graph. This is a problem related to the excanvas and flot JavaScript used to plot the graph. Once a newer stable version of flot is released I’ll update memCron to support that.

  4. Hello!!
    Great application, I was ready to make a nice donation, when I started receiving error emails from Dreamhost:
    "Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /home/xaxa100/portofoli.com/memcron.pl line 40.
    BEGIN failed--compilation aborted at /home/xaxa100/portofoli.com/memcron.pl line 40 (#1)
    (F) You said to do (or require, or use) a file that couldn't be
    found. Perl looks for the file in all the locations mentioned in @INC,
    unless the file name included the full path to the file. Perhaps you
    need to set the PERL5LIB or PERL5OPT environment variable to say where
    the extra library is, or maybe the script needs to add the library name
    to @INC. Or maybe you just misspelled the name of the file. See
    perlfunc/require and lib.

    Uncaught exception from user code:
    Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /home/xaxa100/portofoli.com/memcron.pl line 40.
    BEGIN failed--compilation aborted at /home/xaxa100/portofoli.com/memcron.pl line 40.
    at /home/xaxa100/portofoli.com/memcron.pl line 40"

    Any clue? It would be grateful if you could give a hand..

    1. Hi padelis, the error message you received is related to perl module not found, which shouldn’t happen. Try to see if the following file exist on your server, if not you should ask DreamHost tech support about it:

      /usr/share/perl5/Statistics/Descriptive.pm

      1. I am having the same trouble but all I received from DH support is something about using CPAN to manually install whatever this is. Is this problem common? Haven’t been able to figure out a fix yet.

        1. DH staff installed the statistics module but I am still having some troubles without Crypt::SSLeay. Is it possible to get a list of all the perl modules required by memCron? That way I can ask them to install any and all missing modules at once. Thanks!

          1. Hi Xerotone, it’s interesting that you mention Crypt::SSLeay is required. As you can see, memCron does not explicitly use the Crypt::SSLeay module, it is required for https support by the LWP module.

            The modules used in memCron is as follow:

            • Statistics::Descriptive
            • LWP
            • Mail::Sendmail
            • File::Basename

            All these modules should come as default on all DreamHost servers.

  5. Hi again Yaosan,
    I’ve got another question for ya,
    Say I wanted to use memcron just for the pretty graphs, and not have it actually adjust my PS’s memory. What settings would I need to change in config.cfg to do this?
    Thanks again,
    Keep up the great work,
    -BassKozz

    1. Right now there’s no easy way to do that, unfortunately. If you really want to, go into memcron.pl and find the line that says my $time (line 208), then go to the next line and add:

      $set = 0;

        1. That could actually work, a very interesting use of the configurable value. ;) If my code is robust enough (I hope! :P) then doing that should keep your memory constantly set at the value you wanted without using any of your daily change limit.

          Let me know if that works out for you!

          1. It WORKS :-D!
            The reason I wanted to have memcron operate like this is because dreamhost gave me the first week free, so I want to do some load testing on my new PS and be able to track it (using the Pretty Charts ;-)) w/out having memcron adjust it inbetween/during tests.
            Thanks for the feedback, I should’ve thought of that first.

  6. Yet Another Question for ya Yaosan,
    Are you getting sick of me yet? Just Kidding, that was not the question…

    Can memcron be used to graph and adjust the memory on a MySQL PS?
    If so is there anything I should know about the way the config.cfg should be setup or is it all the same?
    Thanks,
    -BassKozz

    1. I don’t have a MySQL PS so I don’t really know how it works. If it works just like the regular PS, chances are the regular installation will work just fine on that server.

      You need to have shell access and writable directories on your MySQL PS otherwise memCron won’t work. You might also need to create a symbolic link that points to the csv file generated on your MySQL PS to have the graph working.

      1. I contacted Dreamhost’s support team, and apparently they don’t allow you to SSH (shell) into a MySQLPS, so that pretty much kills the deal as far as running memcron on a MySQLPS.
        Bummer :(
        Maybe they’ll add this in the future, but for now I am going to steer clear of the MySQLPS because of this.

  7. How can I avoid seeing Internal Server Errors? I’m not sure what value to set the $mem_free_to_used_ratio variable to. Any advice would be appreciated.

    1. You can increase the $mem_free_to_used_ratio to reserve more memory to prevent internal server error, which happens when you run out of memory.

      You can also set $min_memory to prevent memCron from setting a memory value that is too low for your site. This could be useful if you anticipate the usage to be higher than what memCron predicts.

  8. Thanks but I was hoping for an actual example, what would a higher value for the ratio be? I’m used to seeing a ratio represented like 6:5 or something similar, not just as a single float like 1.2.

    1. Someone PLEASE correct me if I am wrong, but I believe the $mem_free_to_used_ratio is a percentage. So 1.2 is 120% meaning, there should be 20% free above used.

      So for example:
      If you have 100mb used then free should be 20mb total would be 120mb.

      Take this with a grain of salt because I am not 100% sure on this.
      HTH,
      -BassKozz

      1. BassKozz, you’re not quite right there. :P

        As written on the usage page:

        Minimum free vs. used memory ratio you are comfortable with i.e. free memory / used memory

        What this means is if you are using 100MB of memory, and you want to have 120MB of free memory, then:

        free memory / used memory = 120MB / 100MB = 1.2 = $mem_free_to_used_ratio

        As you can see, the higher you set this value, the more free memory you will be reserving. The total memory is just used memory + free memory, and you pay for the total memory you used. On the graphing page, the total memory is actually the “guaranteed” portion of memory DreamHost provides. Add in the burstable portion i.e. swap memory and you have the real total memory usable.

        Back to the question, if you would like to have less internal server error, increase the ratio to 1.6 or even 2. You will notice the difference on the graph after you apply the changes and wait for the new data points to show up.

      2. Couple of minor suggestions for a future revision…
        1. It would be nice if the index.php had a favicon, I usually have multiple tabs open on my browser and it’s hard to pick out which tab I am looking for unless I have a favicon to reference.
        2. It would be nice if the index.php auto refreshed. But it would be even better if the auto refresh was based upon the $cron_interval variable (I currently have my $cron_interval set to 2, and I manually edited my crontab to allow for that), this way it would automatically refresh as new data appeared. Also if this was a new variable (say $auto_refresh, 1 = on 0=off) so that this feature could be turned on or off automatically.

        Just some idea’s for thought.
        Thanks again for all your help and the great work.
        -BassKozz

  9. Great work! Really like its and I look forward to any future development you do on it. One simple thing I would love to see would be a timezone offset. So the timezone could be matched to my current timezone, or even better would be to the timezone of the viewing user. Its minor but would be a nice touch.

    1. Hi Jeffrey, right now the timezone offset is set through a variable found in index.php. I’ll try and see if I could somehow come out with a way to have the timezone automatically detected for the viewing user using JavaScript, that will be really neat for sure!

  10. I’ve got another one (suggestion) for ya,

    The $mem_threshold & $email variables are brilliant, but I would also like to be notified if/when my memory is resized back (BELOW) my $mem_threshold.

    The reason being is that as the code is now we get an email if the memory is resized above the threshold, but we have no way of knowing if/when it has been resized again back to normal levels (BELOW) the threshold. So for example, I recently got a notification that I was resized above the threshold at 1am, so when I woke up the next morning I had to immediately run to my computer and logon to my memcron site (index.php) or my Dreamhost panel to check and make sure that it was back within normal operating levels, to make sure I wasn’t getting charged big $$$ for the excess memory, and ensure that everything was operating as normal. A simple email notification would have saved me from having to do that.

    -BassKozz

    1. Once again, a very good suggestion, BazzKozz. It makes sense to have the notification works just the way you described. I’ll see if I can make it work that way in the next version. ;) Thanks for your suggestion!

  11. Hi Yaosan,

    Thanks for developing this great tool. I’ve been using it for a bout two weeks now, and I’ve noticed that no matter what I set the ratio to, a single measured increase in usage shows a large increase in allocated memory.

    http://img269.imageshack.us/img269/6669/memcronexample.png

    In the example above, for the first half, my ratio was set to 1.05, and as you can see it appears to me that the adjusted memory increases more than necessary for a temporary peak.

    What do you think? Am I misinterpreting the graph?

    1. Hi Ben, the reason for the increase in allocated memory is due to the spike your server experienced, which probably increased the standard deviation significantly. This in turn causes memCron to reserve more memory in anticipation of another spike.

      You can reduce the effect of single spike by either increasing or decreasing the $num_data value. By setting it to a lower value, you reduce the length of time it takes for the spike effect to be completely eliminated. For example, with the default value of 80, the spike will affect the standard deviation for 7 hours. By setting it to a larger value, you can also reduces the spike effect because the majority of normal data point will outweigh the spike effect.

      Another value you can play with is $mem_free_confidence. Lowering it will reduce the effect of standard deviation on memory tolerance, and hence less memory will be reserved.

      1. Got another screen cap for you, Yaosan. http://img189.imageshack.us/img189/7986/memcronexample2.png

        Here are my settings:
        $mem_free_confidence = 0.20;
        $mem_free_to_used_ratio = 1.20;
        $downsize_resistance = 2;
        $num_data = 36;

        So first, I don’t see why the zig zag is happening on the most current part of the graph. The average usage is not increasing so why does it jump back up?

        Second, as you can see I have many approxamately 10% increases in the memory used, but then they drop back down. As long as this doesn’t exceed the memory I’ve allocated, I don’t want memcron to add more because more than likely, it will just come back down. And, Dreamhost lets you burst above your allocated memory anyway.

        Third, if you look at the total graph, you can see that if the allocated memory was set at around 1GB, then the used memory would sometimes touch it, but then would drop back down. This is how I hope we can get memcron to work. …then as less memory was required overnight, it could drop down to around 750MB as well. This is my goal — to better track the minimum memory required.

        So what do you think, are there different settings I should be using, or can you think of a modification to the code that might help?

        Cheers,

        Ben

  12. Hello,

    Is it possible to add a function to set my PS at max (4000 MB) from 7pm to midnight everyday ?

    Thanks a lot !

    1. First of all, are you doing this because memCron isn’t able to scale quickly enough for your server’s need? If that is the case, please let me know more about your server’s usage history and I’ll see if memCron can be improved to accommodate the known spike you are experiencing daily.

      Here’s a possible solution for your question. First, copy your existing config.cfg and name them config1.cfg and config2.cfg. Leave config1.cfg alone and modify config2.cfg such that the max and min memory are both set to 4000.

      Next, set up a cron job that runs at 7pm daily that overwrites your config.cfg, e.g.

      cp ~user/example.com/memcron/config2.cfg ~user/example.com/memcron/config.cfg

      Then, set up another cron job that runs daily at midnight that revert your config files e.g.

      cp ~user/example.com/memcron/config1.cfg ~user/example.com/memcron/config.cfg

      Let me know if this does the trick for you. ;)

  13. I installed it and keep getting the error

    Warning: fopen(/home/dangerlarson/cparedirect.com/memcron/logs/ps11041.csv) [function.fopen]: failed to open stream: No such file or directory in /home/dangerlarson/cparedirect.com/memcron/index.php on line 57
    Can’t open log file!

    I checked & there is no file by that name in what I have on the server or what I downloaded. Is that supposed to be created on the fly & if so, any ideas on what I did wrong. Installation was pretty simple so not sure how I botched that one :)

    1. Matt,

      You missed the most important step of the install:

      9. Apply the settings and wait an hour for memCron to collect sufficient amount of memory usage data

      ;-D

      Memcron needs time to collect the data, it first collects the data in a *.log file then parses it over to *.csv. Check back at your site in about an hour, you should be good to go.
      -BassKozz

    2. Hi Matt, I think BazzKozz pretty much answered your question. (thanks BazzKozz!) Let me know if you have more installation issues. Good luck!

      1. hey Basskozz – actually, my memory is fine. It had been about 3 hours before I emailed you and now it’s been about 7. Still the same error. ;)

        1. Who said anything about your ‘memory’ ?

          Anyways, couple of things aren’t adding up here, your memcron dir/site: http://www.cparedirect.com/memcron/index.php
          Currently reads:

          Warning: fopen(/home/dangerlarson/cparedirect.com/memcron/logs/ps11041.csv) [function.fopen]: failed to open stream: No such file or directory in /home/dangerlarson/cparedirect.com/memcron/index.php on line 57
          Can’t open log file!

          But line #57 of index.php doesn’t have anything to do with the log/csv files, it should be referring to line #85 ($fp = fopen($log, 'r') or die ("Can't open log file!");
          ) for this particular error. Weird
          Couple of questions:
          What version of memcron are you using?
          Have you modified the index.php file in any way?
          Did you make sure to set the correct $key & $ps in the config.cfg file?

          1. - using .3 – will try .4
            - have not modified the index.php
            - $key was copy/pasted right from the dreamhost api screen
            - $ps came from everything before the “.” in my host name “psXXXXX”.dreamhost.com –
            - btw, was referring to memory in my head, not 1′s & o’s memory.

  14. Thanks for the great tool, Yaosan. I’ve already donated.

    Question: in my notification e-mails the running processes section is coming in blank. Something I’m doing wrong?

    1. Hi Jamon, things like that do happen when your server gets very busy, say it is approaching its memory limit or the CPU load gets a little higher than normal for the shell to respond properly. This is also the reason why you will sometimes see a couple of missing data on the graph. Sometimes, the command just didn’t get executed perfectly, and many times you can’t really tell for sure why it happens.

      I assume you only experience that missing running processes section once in a while. If it happens consistently, let me know and I’ll look into it.

      Thanks for the donation btw! :D

      1. It’s definitely every time–I haven’t had one come in with any info yet. Here’s an example:
        Date: Tue Jul 7 06:21:02 PDT 2009

        Total: 300 Used: 276 Free: 24
        Mean: 112 Stdev: 14 Load: 0.00
        Mem: 150 Swap: 150
        Tolerance: 22 Tolerance (free): 363
        Target: 320

        Memory was resized from 150 MB to 320 MB!
        -----
        List of running processes:

        This report is generated by memCron at 06:21:02 AM PDT (2009-07-07).

        1. Okay if it happens every time that something must be wrong. FYI, memCron is trying to run the following command to generate the list of running processes:

          top -b -n 1

          Try logging into your server and type this command in shell. If this works then I can’t see any reason why it returns nothing when memCron tries to run it.

          1. Hm, yeah, from a shell it is working–it returns the list of processes just fine.

            I tried this though and it didn’t work:

            User:

          2. It didn’t post:


            exec("top -b -n 1", $a);
            echo "User: " . exec("whoami") . "";
            print_r($a);

            …hopefully that came through.

  15. Jamon, try running this line in your shell: perl -e '$i =`top -b -n 1`; print $i;', if this works then the process list should work. If not then something is wrong with your shell or perl.

    1. Thanks, Yaosan. It does work and I got a list of running processes. But I just got an e-mail where the list of running processes was blank again. Somehow it’s not transferring to the e-mail.

      Any other ideas?

      1. No unfortunately. One other thing that memCron does differently compared to the code I showed you is memCron will chomp the “top” output. I don’t see how this can make a difference. Sorry, I’m out of idea.

  16. Matt,

    I see your site (http://www.cparedirect.com/memcron/index.php) has changed:

    Warning: fopen(/home/dangerlarson/cparedirect.com/memcron/logs/ps11041.dreamhost.com.csv) [function.fopen]: failed to open stream: No such file or directory in /home/dangerlarson/cparedirect.com/memcron/index.php on line 85
    Can’t open log file!

    The problem is it’s trying to reference a log file called “ps11041.dreamhost.com.csv” but that’s not correct. The file should be “ps11041.csv”. Make sure your config.cfg file reads:
    $ps = "ps11041";
    Also go into your cronjob (DreamhostPanel>Goodies>CronJobs) and enable email notifications (email output to:), and then post back here any errors that get sent to you via email. Maybe that might shed some light on the issue.

    HTH

    p.s. I know which ‘memory’ you were talking about, I never said anything about your memory (in your head or otherwise). I was just kidding around with you when I said you missed a step… Sometimes it’s easy to speed read install instructions, and I thought thats what you did.

    1. The .dreamhost.com was something I just added this AM. previously just had the PSXXXX in there and it didn’t work.

      Removed it & here is the error I’m getting via email.

      Uncaught exception from user code:
      Cannot open memory log file! at /home/dangerlarson/cparedirect.com/memcron/memcron.pl line 119.
      at /home/dangerlarson/cparedirect.com/memcron/memcron.pl line 119

      ideas?

      1. I assume it is probably a UNIX ‘permissions’ problem, but I can’t be sure.
        Can you logon to your shell (SSH) and post the output of the following: ls -la /home/dangerlarson/cparedirect.com/logs/

        Otherwise I am stumped, Yaosan ?

        1. I do think it’s a permission issue. Matt, you can try chmod your memCron’s log directory to 775 just so to ensure the cron running user have permission to create the log file for you.

          Also, make sure you set up your cron job properly by picking the correct shell user that have access to the memCron directory you uploaded.

  17. Yet another suggestion for ya Yaosan,

    It would nice if you kept a log of the target for graphing.
    I am using the “Downsize count” in my graphs to keep track of when I might expect a downsize, but it would be nice if we could also see what the target is for the downsize, currently the only way to see the target is to enable MAILTO in the crontab. Also it would be nice if we could have multiple lines in the graph instead of just two (3 if your using Used+Total Memory). My dream graph would consist of the following:

    Used+Total Memory
    CPU Load
    Downsize count
    Target

    Just some suggestions. Are you getting sick of them yet?

    1. The graphing part can still be improved, no doubt. Initially I want to make all the data selectable via checkboxes for plotting just so you can instantly turn any data on and off. However, since all data has different min, max value and units (CPU uses load while memory uses MB), it’s very difficult to set up the axis right. It’s probably impossible to have used memory, CPU load and downsize count all on the same graph since they all have different units and we have only 2 y-axis usable.

      I can add target to the csv file just so you can pick it from the drop down field on the graphing page in upcoming versions, does that help at all or you’re still stuck with your dream graph? :P

      1. No that would be great if you could add target to the graph drop down list… I can switch back and forth between the different graphs, or maybe just have 2 graphs open at the same time in different firefox tabs.
        I can’t always get what I want ;-P

        Thanks for hearing my suggestions :-D

      2. What about having the Y axis be a % of max? For example, memory would be % of the $max_memory, CPU load would be % of some max load…etc…

        I can see the problems with this suggestion as well.

        One thing that could be a little confusing is when you select two memory Y axis.

        Here’s the query string:
        /?y1=mem_total&y2=mem_free&n=60

        The left and right axis are a different scale, making it difficult to compare the two. Maybe if the two Y axis are both MB they should have the same scale (min and max).

        1. The reason why the 2 y-axis are not using the same scale is because the fluctuation for one (e.g. stdev) can be so small that when you use the same scale as the other axis (e.g. total memory), you won’t be able to see the small fluctuations and it will look close to a straight line to you.

  18. Sweet tool, been running it for a while now on my PS. Any chance you can also log (and graph) the number of connections (netstat -nt) as this will help as a guide to understanding why we are getting spikes in memory usage .

    1. Good idea ozgreg, I’ll probably add 2 versions of netstat output, one with IP and another with hostname into the next version. Is there a way to get both IP and hostname using a single netstat command?

      fyi I’ll be away for about 2 weeks so there won’t be any response and updates from me during this period.

  19. Memcron was working for me before, not sure what happened. Before I had the nice graphs that showed how memcron was adjusting my memory. Now on the graph it appears memcron is doing no memory adjustment though the cron emails claim some adjustment is being done.

    And I’m gettting this in email. (note: I changed the URL and X’d out the values in this comment post.)

    sh: ps10379: No such file or directory
    sh: ps10379: No such file or directory

    Date: Tue Jul 21 00:31:10 PDT 2009

    Total: 700 Used: 495 Free: 205
    Mem: 350 Swap: 350
    Mean: 270 Stdev: 68 Load: 0.26
    Tolerance: 129 Tolerance (free): 513
    Target: 558

    Memory was resized from 350 MB to 558 MB!
    Uncaught exception from user code:
    Error at https://panel.dreamhost.com/api/?key=&cmd=dreamhost_ps-set_size&unique_id=8081e7d3-b31c-48ab-a84a-2c7cd02239cb
    &ps=&size=558
    401 Not Authorized
    Aborting at /home/hawaii247org/hawaii247.org/memcron/memcron.pl line 272.
    at /home/hawaii247org/hawaii247.org/memcron/memcron.pl line 272

  20. Guessing from your 401 error that your API key is incorrect, suggest you might want to verify the DH API key again (or generate a new one)..

  21. I have a MySQL PS and I would like to try to get this working on it as well. Since you can’t SSH to the MySQL PS, is it possible to monitor the current memory usage/load via the DH API instead and react to that instead? I’m willing to test this solution if you like.

    1. This is a tricky proposition, because that would require giving memCron the user&pass to your dreamhost Panel, which is a security concern. As it stands right now memCron only has access to the API and can only run API calls that you’ve enabled when you create the key. Allowing memCron to access the Panel is a big no-no as far a security is concerned because with that information someone could do some real damage to your sites/domains/etc…
      While the config.cfg file is protected with .htaccess (i.e. you can’t view the file from http://memcron.domain.com/config.cfg) if the .htaccess file were to be mistakenly deleted this would open you up to the possibility of someone gaining access to your DH Panel’s user&pass. :O
      Not only that but memCron is currently only setup to handle SSH commands not webscraping, so it would be a MAJOR undertaking to scrape the DH Panel, and if DH ever changed the way they reported information on their Panel it would break the whole thing… Not worth the effort IMO.

      A possible alternative would be to use the api’s [code]dreamhost_ps-list_usage[/code] call to determine the MySQL PS’s usage, it is unclear how often this information is updated, but from what I can gather from the wiki: http://wiki.dreamhost.com/Api#dreamhost_ps-list_usage
      This gives the results of the past 30days in a daily AVERAGE of the amount of memory used. Therefore it might be possible to setup memCron to run on a “daily” basis to check the previous day’s average and adjust accordingly. Not that great of a solution compared to what memCron is currently setup to do (run every 5minutes and adjust accordingly), but I think it’s the best alternative available at this time.

      I would submit a request to Dreamhost and ask them to change the [code]dreamhost_ps-list_usage[/code] call to return the CURRENT usage statistics instead of a daily average. Maybe if enough of us request that as a feature DH might make a change.

      1. If there’s a way to get say 5 minute average usage via API then it shouldn’t be hard to adapt memCron for MySQL PS. However, I do not have an MySQL PS myself so I might need a demo access to test the code out. Anyway, it will probably be some time before this API feature gets implemented, if ever.

    1. David,

      Two things:
      1. Can you logon to your PS via SSH and run paste in the following:
      ls -la ~/iiichan.net/stuff/memcron/logs/
      Then paste the output here.
      2. Go into your DreamHost Panel>Goodies>Cron Jobs, and modify the memCron cronjob and put in your email address in the Email output to: field, and report back any errors you get via email.

      -BassKozz

      1. total 12
        drwxrwxr-x 2 humblefool pg198835 4096 Jul 22 17:22 .
        drwxrwxr-x 6 humblefool pg198835 4096 Jul 22 18:48 ..
        -rw-r--r-- 1 humblefool pg198835 49 Jul 22 18:48 .htaccess

        I already turned on email reporting for the cron job, and haven’t received any emails.

        1. running ‘perl /home/username/iiichan.net/stuff/memcron/memcron.pl’ generates a file and the correct memcount, but the cronjob doesn’t seem to be functioning. Hmm.

  22. Would it be possible to modify the program so that it could break down the chart display to show which domain the CPU and memory load is coming from? I have several domains running on the same PS and I would love to be able to chart the usage for each domain rather than just the overall?

    1. If this were possible and were built into memcron I’ll donate again immediately. That would be so helpful. Not sure if it’s possible though.

    2. I highly doubt that this is possible, as the processes that are running on the PS don’t differentiate between domains (or subdomains).
      Yaosan might know?

        1. Good point… Didn’t think of that.
          That might do the trick however, if your using mod_php, this won’t work as all the processes are run under the user “dhapache”. Not sure if the same holds true for standard PHP, as I haven’t tested it.

        2. Also there are some other processes that run under:
          root, postfix, dhapache, daemon, etc…
          That wouldn’t be able to be directly tied to a specific domain (or subdomain).

          1. True. But those could still be listed separately and you could debug from there. I’d definitely like this functionality.

          2. Sure, BassKozz, I think it would help a lot of us. My PS has taken an average 100 MB jump in the last 10 days for no apparent reason and I want to find out why. I don’t know what was active in the top command before the jump so I can’t compare very well. Rebooting has no apparent effect.

        3. It won’t be that easy to track usage by shell user. The only way I can think of is parsing the top output and average out the usage over say one minute for each user, and use that to assume how much memory is each user using.

          It will take some testing to just get that data properly stored in csv and plotted using PHP and Javascript.

  23. I installed memcron about a weeks ago. It has been working relatively well so far. A nice side effect of the graph is that it allowed me to catch a problem with the cron jobs. I’m really looking forward to see a 1.0 version.

    1. It also helped me catch a problem…
      For some reason on 7/25 my cronjobs stopped dead in their tracks, I tried to “crontab -l” and got a “permission denied” error. Long story short my permissions get fudged up some how. I contacted DH support and they fixed it, but they couldn’t explain why this happened in the first place :(
      Anyways, I am curious to hear what problem you ran into Joe?

      1. I don’t know what the problem was as I found out a couple hours after things went back to normal. I contacted DH support and while they saw that something went wrong, they couldn’t figure out why…

    2. I agree that the graph is very useful to find out if your server is down, which will show up as missing data points for a period of time.

      From the graph, I also notice at around 9:15pm EST yesterday, extra swap memory is allocated to your server for the same amount of memory set via API. I believe DreamHost might be rolling out an update to allocate extra swap memory to handle spikes better for customer that is not running memCron and is having trouble with internal server error due to spikes.

      As well all know, memCron is already doing a great job without the extra swap memory. If this update is permanent, memCron can operate at a lower margin, and you will be able to save even more money with it. :D

      1. I think DreamHost is rolling out a new update that gives extra swap memory to your PS server, this is the reason why you see the jump right after memCron changes your memory setting, signifying more memory is available than what memCron expects.

        If this DreamHost PS change is permanent, I will update memCron to adapt to the new memory settings.

        I also notice you can now create a root user for your PS via DreamHost panel.

        1. I’m not a Perl programmer (PHP only) but I noticed in memcron.pl on line ~339 it says:


          } elsif ( $mem_target > $max_memory ) {

          Shouldn’t it be:

          } elseif ( $mem_target > $max_memory ) {

          ? That’s probably the issue.

  24. Unfortunately, this doesn’t work in Firefox 3.6a1pre. The graph is not shown, the text is not styled and in the error console, it says “Error: $.plot is not a function; Line: 158″

    I don’t know whether the problem is memcron or Firefox. Could you please have a look at that? Thank you very much!

  25. I’m getting a lot of memory spikes and plateaus.

    http://www.devcsd.com/memcron/

    What do you suggest? I changed a few things to combat it less than an hour ago but it’s still doing this.


    $max_memory = 600;
    $min_memory = 250;
    $mem_free_confidence = 0.992; // was 0.999
    $mem_free_to_used_ratio = 1.2; // was 1.5
    $downsize_resistance = 2; // was 6

    1. Same here, started at roughly 1:30am EST for me.
      I did some testing and found an Odd result… I set $max_memory and $min_memory to 150MB. Yet I am still getting these results:
      http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-bizarre.png
      http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-bizarre2.png
      http://i4.photobucket.com/albums/y105/basskozz/memCron/memCron-bizarre3.png
      Also according to my DH Panel, memCron keeps changing the memory size, to 150MB, even thou it’s already there:

      2009-07-31 05:57:57 150 MB $15.00 29 mins 49 secs $0.0100
      2009-07-31 06:27:46 150 MB $15.00 29 mins 34 secs $0.0099
      2009-07-31 06:57:20 150 MB $15.00 31 mins 12 secs $0.0105
      2009-07-31 07:28:32 150 MB $15.00 28 mins 59 secs $0.0097
      2009-07-31 07:57:31 150 MB $15.00 29 mins 51 secs $0.0100
      2009-07-31 08:27:22 150 MB $15.00 30 mins 52 secs $0.0104
      2009-07-31 08:58:14 150 MB $15.00 29 mins 19 secs $0.0099
      2009-07-31 09:27:33 150 MB $15.00 29 mins 55 secs $0.0101
      2009-07-31 09:57:28 150 MB $15.00 29 mins 56 secs $0.0101
      2009-07-31 10:27:24 150 MB $15.00 29 mins 47 secs $0.0100
      2009-07-31 10:57:11 150 MB $15.00 30 mins 8 secs $0.0101
      2009-07-31 11:27:19 150 MB $15.00 9 mins 11 secs $0.0031

      Seems like a Serious problem with memCron and/or the DreamHost API.

    2. As Yaosan said above, DH is probably rolling out some changes which has caused memCron to go haywire. I’ve disabled memCron on my PS until this gets straightened out.

  26. Something strage is heappening with my memcron: In the graphs shows that I have 225Mb of memory, when I actually have only 150Mb.

    Then I’ve manually change to 188Mb and now the graph shows 280Mb

    Something is not OK, I’m gonna reinstall it. Just to let you know of this problem.

    1. We had this discussion before, and the conclusion is it is not possible for memCron to run on MySQL PS unless DreamHost starts providing API to access instantaneous memory usage on MySQL PS.

  27. It’s hard to find your articles in google. I found it on 13 spot, you should
    build quality backlinks , it will help you to get more visitors.
    I know how to help you, just type in google – k2 seo
    tricks

  28. I see you share interesting stuff here, you can earn some additional cash, your blog has huge potential, for the
    monetizing method, just search in google – K2 advices how to monetize a website

  29. For me, choosing a meicadl specialty is easy. I was lucky enough to have the opportunity to work as a Dermatologist Tech prior to moving to Alaska. What an amazing field! No two cases are the same and more than one type of procedure can be performed in a day. Working for a dermatologist provides the benefits of both a private office setting and hours along with the experience of surgeries and unique treatments. Cyst removals and cancer treatments became the most interesting procedures to me and I can not wait to get back into the field!There are not many positions or specialties I would turn down when it comes to the meicadl field. If I had to pick one, however, it would have to be gynecology. Why would this be the field I turn down? I am honestly not sure. I must be lacking the gene the would cause me to be interested in the subject! For whatever reason I would choose to work anywhere else before choosing gynecology. Specialty clinics are like sushi I guess. How do you know you don’t like it if you’ve never tried it?!

  30. JOIN TEAM JOEY NATION!BUFFALO WILD WINGS IN AUBURN, IN Wednesday NOV. 14th from 6-8(Come on out!) You may contact for couopns. 10% of all food purchases will be donated to help JOEY!**TEAM JOEY Christmas Extravaganza** Mark your Calendar!!Date: November 17th, 2012Time: 12:00pm 5:00pmLocation: Auburn First Christian Church,910 N. Indiana Ave. Auburn, IN (Just south of Rieke Park) SANTA will be there for a photo Opp, bring your camera, take all the pictures you need and make your Christmas Cards perfect, coloring table for kids to color pages for the Keller’s, Party lite candles, Premier Designs, a Certified Massage Therapist (for on the spot massage), home made Jams, bake sale, 31, Vault Denim, home made blankets, scarves and purses and many more things! SILENT AUCTION: Debrands chocolates gift baskets, American doll clothes, Cookie cottage gift baskets, Italian Grille gift cert, Albright’s Grocery gift Cert., movie baskets, longaberger baskets, Vera Bradley Bags and MUCH MUCH MORE! Please join us in making this a SUCCESS FOR JOEY!!!! Please share and mark your Calendar!

  31. Breakfast with Santa! Saturday, December 1st from 9-11AMOssian United Methodist Church201 W. Mill Street in Ossian, In 46777Come have pancakes, deiuciols egg casseroles, smoky links, cereal, donuts, dutch crunch dessert, Coffee, milk and juice something for everyoneHave your children’s picture taken with Santa and then they can shop in the Elf Store for their family members. Elves will be available to help them shop so they can keep it a surprise! All gifts are $2 and gift wrapping is included. This is a fun, holiday event sponsored by the Norwell High School Show ChoirSee you there!

  32. I recently moved one of my blogs that was on Blogger (gasp!) to WordPress.org. The prelbom was that I had built the blog on Blogger to the point of getting hundreds of visits per day from Google Search and didn’t want to loose all that SEO traffic. Blogger doesn’t allow you to create Domain Redirection the same way that wordpress.com does.To work around, I setup a CNAME for my blogger blog (wordpress.com calls this domain mapping ). Then I waited about 3 months for Google to re-index all my pages with my new custom domain instead of my Blogger subdomain. (You can check to see how many pages Google has indexed from your site by searching site:yourdomainhere.com ).Once my new custom domain was fully indexed, I made the transition following similar steps to what John listed in his post. Glad to say that it worked and I now have all my search engine traffic going to my blog on wordpress.org.I hope this helps someone else who feels stuck on Blogger. You can do it, it just takes time and a little work!

  33. Andrew, I really like your blog! I built my blog three mthnos ago with ST. I’m using the theme with no mods to the code at all, and the blog is already averaging 300 400 page views a day, and growing at 15-20% each month. That growth is over twice as fast as the blogger blog I built on the same topic. I’m no expert, but the only thing I can attribute that kind of growth to (besides awesome diy posts!) is my well-spent money on Standard Theme.P.S. I’m not an affiliate for ST, so my glowing endorsement is unbiased.

  34. An impressive share, I simlpy given this onto a colleague who was doing somewhat evaluation on this. And he in fact purchased me breakfast as a result of I found it for him.. smile. So let me reword that: Thnx for the treat! However yeah Thnkx for spending the time to debate this, I feel strongly about it and love studying extra on this topic. If potential, as you turn out to be expertise, would you mind updating your weblog with extra particulars? It is highly useful for me. Large thumb up for this blog post!

  35. With havin so much written cnnteot do you ever run into any problems of plagorism or copyright violation?My blog has a lot of exclusive cnnteot I’ve either created myself or outsourced but it appears a lot of it is popping it up all over the web without my permission. Do you know any methods to help protect against cnnteot from being stolen? I’d definitely appreciate it.

  36. Thanks Nathan,I am with ForeFront web, a company that spailecizes in (but we work with firms all over the US and the world). We are experienced in developing sites of all sizes, and are experts in PHP, Joomla and WordPress, eCommerce sites, SEO services and social media consulting. Oh and we rarely wear socks. Feel free to contact us for a !Cheer

  37. Hi John! I recently mitgerad from wp.com to wp.org.. One bizarre issue I cannot seem to resolve: when viewing posts via monthly archives, there’s a redundant listing of 2 Replies and 2 Comments in the footer of each post. Do you have any idea why this is happening and what the resolution might be? Thank you so much!!

  38. I loved up to you’ll receive cirared out right here. The comic strip is attractive, your authored subject matter stylish. nonetheless, you command get bought an nervousness over that you wish be turning in the following. ill no doubt come more beforehand again since precisely the similar nearly a lot regularly inside case you defend this hike.

  39. John,Thanks for the great heads-up on Standard Theme. Do you have any suggestions on fidnnig persons to develop a relatively simple site/blog with Standard? I’ve tinkered with WordPress as well as Headway for WordPress so I could do it myself, but I would really like something that looks more professional.Thanks for all you do!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>