memCron 0.4.1 fixes memory issues

memCron has been updated to address the 2x increase of swap memory recently implemented by DreamHost. So far, there hasn’t been any official confirmation of this change. I’m guessing DreamHost is still experimenting with this new change and doesn’t want to make any commitment to support these extra swap memory just yet.

Here are the changes I’ve made in this minor update:

  • Added: favicon for easier recognition of memCron tab in browser
  • Fixed: Memory calculation due to the 2x increase of swap memory by DreamHost
  • Fixed: Proper max axis value for “Used + Total Memory”

Download: memCron 0.4.1 (32.57 KB)

Please give this update a try and let me know if there’s any issue. I have tested this on my own PS server and it seems to work just fine.

198 thoughts on “memCron 0.4.1 fixes memory issues”

  1. yeah memcron is NOT working properly for me anymore.

    it sets the memory to around 2427 MB when I’m only using around 300-400mb on the DH panel.

    I had to disable it.

  2. From DreamHost customer support:

    Sorry about the troubles you’ve encountered. As for the weird memory readings you’re reporting, there’s actually a valid, albeit weird reason for this. Previously, we had no way to track cache usage on a per-guest/PS basis. With a new kernel update that’s rolling out, it can now be accurately reported by the guest. In light of this, the guest’s kernel now reports cached memory as well, where before it was always 0.
    Now this is being included in the free counts, yet this cached memory can be reclaimed by the system at any time.

    Currently, all memory readings do look fairly accurate for the output of ‘free’ in light of the changes:
    ps1****:~# free -m
    total used free shared buffers cached
    Mem: 912 258 653 0 0 0
    -/+ buffers/cache: 258 653
    Swap: 0 0 0

    As for your reference to
    http://memcron.com/2009/08/memcron-0_4_1-fixes-memory-issues/#comment-342
    yes, all PS guests are under a trial to get rid of the swap reporting, since in reality, SWAP doesn’t really exist on a PS but is instead managed by the host. As for how the numbers now work out, I’m afraid I can’t really comment too much on that as well nothing’s been really announced yet.

    1. And now DH support just emailed me again saying there’s something wrong with the mem usage being reported on my account. They’ll “get back to me” *sigh*

    2. Here is an updated version of memcron.pl* that calculates memory usage by summing stat information from each individual process. It also takes into account the new kernel by ignoring swap when the kernel is reporting cache. I hope it helps those who were getting wild numbers.

      * This isn’t my new version with the separate collector daemon.

      1. Hi Ben, I replaced my memcron.pl with your source code but my memory is still resized to 4000 MB, following are the command result:


        Total: 450 Used: 429 Free: 21
        Mean: -3602879701896090 Stdev: 7205759403792756.00 Load: 0.06
        Mem: 150 Swap: 300
        Tolerance: 13667009612242782 Tolerance (free): 22267470489563948

        Target: 4000

        Memory was resized from 150 MB to 4000 MB!
        API Response:
        success
        memory_mb
        4000

          1. This one is working well for me. It is reporting very close to the same numbers I see on DreamHost’s resource usage charts. Tomorrow, when it’s no longer taking the corrupt history into account, I should be able to let it start managing memory again.

            Thank you!

  3. The problem started a few days ago, my memory usage was creeping up above normal. And since yesterday, it always jumps to 4000 MB. Now I have to disable it.
    Is there anyone has a workaround for this?

  4. I also disabled memcron for the time being, as it was way off. I contacted DH support and now I understand why. I will try that .pl you link to when I have the chance.

    I don’t want to tell you how to run your site, but you should put a notice on the front page here, as I stopped by and when I saw there were no new posts since last year I figured the problem was not with memcron.

    Regardless, I think memcron and DreamHost PS are great. The more I use them, the more I like them.

  5. I’m not getting any email notifications when memory goes past the threshold. Must I use my domain name branded email? I wanted to put in an email that would be sent to an email account that would push to my phone.

  6. I think we need to add a way to drop one or two spikes off the memory usage before adjusting memory like psmanager does. It doesn’t need to adjust memory way upward for a one-time spike in usage and then wait an hour before it can downgrade it again. I might look at that myself next week if no one else wants to tackle it. And then, the internal processing in memCron is handled differently so it may not even be possible. Time will tell.

    1. My new version (which is so far off of memcron that it’s hard to call it that) uses different algorithms to handle noise as well predict the ebbs and flows throughout the day.

      I’ve got it tracking the memory usage of daemons, scheduled (e.g. cron/at), and interactive processes separately and based on that can make some (usually valid) assumptions about whether a spike in memory usage is transient.

      Been wanting to get it released for a few weeks now, will try for next.

    2. That sounds great actually. I’ll just sit tight and wait for Ben’s new memory manager.
      And now I have another problem. Yesterday I was trying to figure out why the memory was being set to the max from time to time, so I put code in memCron to set the downsize_count to -1 right after the call to the API to set memory. Today, I’m looking at the log and when it jumps to max, the number is not -1, so apparently, something else is adjusting the memory. Time to create a new API key and revoke the old one I guess. I wish you could add a comment to the adjustment so I could see WHAT was doing it.

      1. OK… maybe it IS setting the memory. Looking more closely, there are generally 300 second gaps between log entries, and where the memory spikes up, there is 600 seconds, so perhaps it’s crashing before it can log the result. :( Sigh.

      2. OK… maybe it IS setting the memory. Looking more closely, there are generally 300 second gaps between log entries, and where the memory spikes up, there is 600 seconds, so perhaps it’s crashing before it can log the result. :( Sigh.

        Now I see. It’s probably getting a bad return from the API call, at which point, it dies rather that alerting anyone. This should be sending an email to alert someone if it can. Certainly it shouldn’t be calling die at this point unless you’re debugging.

  7. Ben-

    Thanks a ton for posting that fix, greatly appreciated. I noticed my last DH bill was unusually high, but didn’t really think anything of it. It wasn’t until I came here that I realized there was a problem with the script. I installed your 0.5.3 version last night and I can already see the difference this morning. The memory slider is much closer to where it needs to be than before.

    I’m looking forward to the new version you’ve mentioned in Feb. 26. Can we officially call this a fork of the original product? If so, I would be happy to help you set up a website to host this project — just let me know. It certainly deserves more attention than to just be buried in the comments of this blog post ;)

    Thanks again for your hard work!

    -Richard

  8. Noticed our memcron graph doubled when I woke up this morning. Checked the DH panel and noticed a HUGE drop in our memory & CPU usage. PuTTY’d in and checked htop and saw strange stuff.

    Live chatted DH support and they said that they’ve doubled memory for PS servers or, more accurately, they halved the price and removed burstable swap memory. I think memcron might need a new version (I think I’m still using 0.41 though).

    What I don’t understand, though, is why the sudden change in graphed usage if this is just a price change and I’m hoping more experienced and knowledgeable people here might have an answer. If it is just a price change, shouldn’t graphed usage remain the same at least on the DH panel resource usage graphs (in Manage Resources)?

    I can maybe understand memcron graphs being off because memcron assumed 900 MB of burst swap memory but not the DH panel graphs. A website I’m hosting that usually goes from ~800-1800 MB of memory usage (set at 1000 max with memcron) suddenly dropped to 50-500 MB. Site traffic should be the same so memory shouldn’t drop that drastically for the graphs to be accurate, right?

    I’ve also previously monitored memory and CPU usage using htop which seems to be different now as well. htop shows ~160 MB usage out of 1000 MB and now several of the bars on the meter are colored yellow, which I’ve never seen before. (I never understood what the occasional blue bars on the four CPU meters meant either). However, when I use top only, the memory usage is ~680 MB at the moment?

    Something changed and everything I’m used to no longer makes sense to me. Anyone want to help enlighten me? Cheers.

    1. I’m now using Ben’s version 0.5.3. I also figured out what the colored bars meant on htop so, uh, nevermind that. Heh.

      Talked to DH live chat again and basically they said that the panel graph should now more accurately represent how much memory a PS is using that is directly affected by how much memory we allocate.

      As for memcron or, er, Benscron, when my memory was set at 1965 MB, total memory on Benscron read 1065MB. When I switched back to 0.4.1 memcron, it also still reads 1065 MB.

      1. Right — the multiplier in get_mem_size() is wrong now that there is no longer any burst memory. The quick resolution is to replace that sub with:

        sub get_mem_size { return (300, $_[0] – 300); }

        The numbers are much more straight forward, but make no mistake: at the low end DH has actually reduced your total available RAM by 150 MB for the same price. They used to provide 450 MB (150 base + 150 burst + 150 system/bonus/unclassified). I treated that extra 150MB as memory to be used for cached data or the dozens of cron and at jobs that DH schedules to run on your private server throughout the day. So now, if you’re using a lot of real memory, first your cached data will be released, and then processes will be killed.

        At the higher end they were overcharging for memory anyway, so it’s all good!

        I’ll be putting a new version up on github eventually — but for now my 0.5.4 of the old memcron is here: http://ben.pastebin.com/jSgVG8fp. Give it a try, and let me know if you’re having any trouble.

        1. I can’t find the sub get_mem_size in the latest version 0.5.4 you linked to above, which I’m using.

          At this point, the memcron graph still doesn’t match the DH panel graph and DH tech support is saying to allocate according to the numbers on the DH panel graph. As I understand it, their new changes (“newer kernel”) allow DH to now track the “cache number”, which is now visible on shell access through, say, free -m, top, or htop. The DH panel graph shows the “used” number minus the “cache” number, and supposedly we’re supposed to allocate memory according to that final number.

          In my situation, this seems to result in massive savings. Not only is the price per megabyte of memory halved, our recorded usage is also much lower than previously. Previously, our DH panel graph showed memory usage anywhere from like 500-2000 MB, that spike being larger than the maximum memory I (as someone who doesn’t understand all the technical stuff) thought was available (ex. I set 1000 MB and there’s 900 MB of swap/burst memory, so how could the PS use more than 1900, right?). As you say, now the numbers are more straightforward but I’m still reeling from the sudden difference not only in price but actual memory used. I’m trying to understand how we were using so much more memory before. Was it that there was cache before but it just wasn’t shown? Meaning, we had more memory than we thought before but were allocating more than necessary because we couldn’t see that extra memory? Doesn’t that mean we were paying more before?

          Anyway, I’m seeing <100-300 MB usage on the DH panel graph now, while I've kept our memory set at 1000 MB. That suggests we have an overkill wiggle room of 700 MB. the 0.5.4 memcron graph shows 484-1319 MB memory used, with 333.3 MB of total memory. Clearly these don't match up due to the changes but what I'm worried about is mostly whether or not memcron can still be depended upon to adjust memory right now. I'm guessing not, right? Because all the numbers and calculations are wrong.

          Would it be smart for us to disable memcron for now and manually lower our memory allocation down to say 300-400 MB to save money?

          Guess we'll have to wait for you to come to our rescue. Whatever happened to Yaoshan the original developer anyway?

          1. The one posted yesterday (0.5.4) is changed already — the replacement was just if you just wanted to duct-tape your current copy.

            I think with you being on the 0.4 series, memcron began charting and adjusting your memory incorrectly when DH started tinkering with these settings a couple of months ago. It sounds too like your min memory might be off and/or the other thresholds.

            Let me just say that I’m not faulting DH for fixing their billing to match their intent with the slider. At the low end they used to give you more than you paid for, they’ve corrected that now but done it under the guise that you’re getting more for the same price.

            For example, under the old method, you paid $15/mo with the slider at 150 and you received 450 MB of usable memory (processes were only killed when your actual memory usage exceeded 450 MB). If you wanted 600MB (now $30), you adjusted the slider to 240 MB ($24 then). For 900MB ($45 now), the slider was at 360 MB ($36 then). And so on until you hit 450 MB on the slider where the “swap” was then capped at 900 MB — that was the best value 1350 MB of memory for $45 vs $67 today. Beyond that the prices converge again, up to about 2GB where the new billing model starts to save you money (4GB used to cost $310 vs $200 today).

            I just realized that the memcron webcharts are wrong because the PHP file is using the old memory formula too. I’ll see about uploading a patched copy, unless someone out there wants to be ambitious.

            No idea what happened to Yaosan — but I am going to fork this project over at github.

  9. Greetings,

    pardon my bad English, and I hope I can understand enough to help me.
    I do not know very much about these technical things server utilization, memory, etc. And so I’m pretty lost with all these changes in DH.
    Trying to understand your comments, I am clear that this new version 0.5.4 of Ben (Many thanks for your hard work;)), tries to reconcile these differences. The problem is that I’ve already updated, but still large differences between DH and Memcron. For example in figure DH Memory usage: memcron one 124MB and 450MB Used Memory of a total of 845MB Memory !!!..
    Why the difference? ..

    Is it because I just updated the file memcron.pl, and I have not changed the parameters set reference Yaosa version 0.4.1 (default)? What are the recommended parameters (default) to use in this version 0.5.4 or future?

    Thanks for your responses and please forgive my “Tarzan English”: (

  10. I got an email from someone working at DreamHost before this doubling of memory is announced in the newsletter. I’ll try to spend some time this weekend to get a new version out.

    btw thanks Ben for addressing some of these issues for some of our users. I did not experience any problem with 0.4.1 myself but seeing how some users are experiencing isolated issues, I tried to get someone at DreamHost to give me heads up about all these hidden changes they are making on some of their PS servers. Unfortunately, I haven’t heard anything back from them yet.

    Lastly, I don’t think there’s a need to fork this project. I still have lots of plan with memCron and if you would like to help, you can send suggestion etc. to me via .

    1. 0.4.1 worked fine. My initial set of changes was to eliminate dependencies and exec’ing processes. Some problems came up from the tinkering that Dreamhost was doing prior to settling on the now current configuration, but those may or may not have affected you depending on your usage pattern.

      You’re right, forking probably isn’t necessary. I mainly wanted to put the project on github.org so that bugs and patches could be tracked better.

      The 0.5.5 version here: http://ben.pastebin.com/qq2NmUEU should work fine for everyone – although the charting page still needs to be updated.

      1. I might skip a release to 0.6 and incorporate those features and fixes you added to 0.5 if you don’t mind. I’m open to the idea of putting this project on github, or I could host some sort of private svn under this domain. We’ll see.

    2. I have a few changes I make every time to this, mainly for my own use. When things go crazy at DreamHost, which they have several times in the last six months, I like to be able to continue tracking memory without memcron actually adjusting it. So I add a feature that lets me disable updating in the config file. I also added mem_target to the csv file so I could track that. Then recently I’ve noticed that when things seem to go haywire, there memcron adjusts memory and then doesn’t log it. The reason? The DreamHost API returned something invalid yet processed the request. So, I modified the return code check to send me an email rather than simply calling “die” which gives no indication to anyone that something bad happened. Don’t know if the code will come through ok, but here it is:

      if (!$response->is_success) {
      $message .= "Error at $url\n ".$response->status_line."\n";
      if ($email) {
      my $top = `top -b -n 1`;
      chomp($top);

      open( MAIL, "|/usr/sbin/sendmail -t" )
      or die "Unable to open sendmail: $!";
      print MAIL <<"ENDMESSAGE";
      To: $email
      From: memCron
      Date: $date
      Subject: Memcron API error on $ps

      $message

      -----

      List of running processes:

      $top

      This report is generated by memCron at $date.
      ENDMESSAGE

      close(MAIL);
      $message .= "\nNotification sent to $email.\n";
      }
      } else {

  11. Another update… now that the numbers reported by /proc/meminfo are better, this does a fine job of tracking memory.

    I’d recommend looking at your config.cfg, in particular at $mem_free_confidence and $mem_free_to_used_ratio. Both values at 0.5 is probably a good place to start, and you can test the configuration after updating it by running “./memcron.pl -d”.

    Enjoy.

    1. I’m still hoping for that fix that drops the occasional spike rather than adjusting memory way up and leaving it there for an hour. I had that feature when I was running psmanager, and I really, really miss it.

      1. I think I can tweak the algorithm to add in spike detections and adjust the $downsize_resistance accordingly if the spike isn’t followed by more spikes. An interesting problem to solve, I’ll look into it.

      2. Also keep in mind that there is no more burst. So the largest spike Dreamhost will accommodate is one that eats up your remaining free memory and then causes all of your cache to be flushed up to the limit you’ve set on the slider.

        I’ll get my daemon code updated and released as soon as I can. It collects memory stats in near real time so it can detect spikes earlier. Even then it still takes another couple of minutes before a resize takes effect….

  12. Hey Ben,

    Thanks again for all the great contributions you’ve made to this fork. I realize you’re a busy guy, and in hopes of making best use of your time, I’ve set up a homepage to keep track of new releases of this project. Kai coined the name “bensCron” and thought it was very clever. So, bensCron is born:

    http://benscron.com

    Ben, if you’d like, shoot me an email at support (at) benscron (dot) com and we can figure out a system for getting new releases up on the site. You mentioned setting up the fork on GitHub and we can most definitely link to that as well. Also, I can set you up with an account on bensCron.com if you want to make updates yourself.

    I don’t want anything out of this other than seeing this great project stay alive. Hope you don’t mind that I registered that domain, but I figured what better way to get the ball rolling on this than to jump right in. Looking forward to talking soon.

    -Richard

      1. I *just* saw that Yaosan is active again with the project after posting my last message. I think it is great! Ben I agree, benscron.com may no longer be necessary.

        For now, I’ll keep announcing the latest releases of both projects at http://benscron.com. And if the two projects end up merging back into one at some point in the future, even better.

        I’m very grateful to you both for your efforts on these projects.

        Best,
        Richard

  13. Whatever you wonderful folks do, I’m totally awaiting a sorted out update soon so I can stare at a pretty graph that makes sense relative to what the DH panel graph says. It kinda sucks having memcron on but not being able to easily tell if it is working…but I suppose that’s partly because I’m still uncertain how the new changes affect how I should allocate memory. For example, sometimes on free -m, I’m seeing no swap and now I’m seeing 600 MB of swap. I guess it’ll take time to figure out all these things again.

    1. Swap is no longer burst — it’s just a mechanism for adding memory to the vserver. When a resize happens, you may see either “Mem” or “Swap” change — but the “Total” (as reported by “free -tm”) will be the value on the slider.

      Upgrade to 0.5.5 here — http://ben.pastebin.com/qq2NmUEU Yaosan is going to release an update to the charting page soon I’m sure.

  14. Yesterday Dreamhost replyed to my support request (33207468) the following.

    We’ve recently made some changes to how our memory reporting works, so
    I’d recommend contacting the dev as to when he’ll be updating memcron to
    work with those changes.

    Please see how Memcron works on my PS:
    http://www.sdtv.gr/dreamMemcron/

    Thank you for your effort.

  15. Ben I Installed the version u put out on the 12th of march and since then memcron hasn’t been working/adjusting the memory. :/
    I have the .55 version installed

  16. Actually it still is adjusting the memory but it reduces it down to 300′mbs
    but it was pinned at 300mbs (lowest) for two days
    you mean to tell me there were no spikes that required more than 300 mbs?
    I doubt that. I’m gonna revert to .54 and see what happens.

    1. 300MB is the minimum, and from what I’m seeing both on my memcron graph and on DreamHost’s own graph, it’s very realistic that you’re not using more than that any more. Of course, nothing says that DreamHost won’t “fix” that if they see their income sink. I think they stopped counting Apache memory or something.

  17. It looks like this last round of changes by DreamHost dropped a bunch of stuff out of how they calculate used memory. 2 months ago I was running between 600MB and 1000MB. I moved some stuff off–all my media files got put on a new PS with nginx, and most of my WordPress sites got moved off DreamHost altogether. Then my memory was running in the 250-350MB range most of the time. After last week’s change, now I’m in the 50-75MB range with no changes on my part. Very strange. Now my “big spikes” hit around 100MB instead of 1000MB. Gotta love the cost savings even if it is rather confusing. Memcron 0.5.5 (ala Ben) with my usual additions is working great here now that spikes aren’t going wild like they used to.

  18. For some reason neither Benscron 5.5 or hacked memcron will change my resources anymore? It’s drag to have to keep monitoring the server for spikes, then to crank up RAM.

    Am I doing something wrong, or does anyone have another solution?

    1. Make sure your API authorization key and machine name are both correct. Without those being right the API call will fail and the official version will simply NOT report it to anyone.

      Another thing you could do is have the output emailed to you, then scan through them to see why it’s failing. Gotta say that was very painful as you get an email every 5 minutes when you only need the one failure, but that’s how I initially tracked it down on my end.

  19. So last night DreamHost added 7640MB of swap space to my PS. That lovely move makes the memcron graph pretty much unreadable.
    http://wcgweb.org/memcron-20100319.jpg
    Sigh. Now it looks like I have tons of memory and am using nothing. I may need to exclude swap from the graph to make it readable again…

  20. OK, I’m getting tired of making the same changes over and over, and there’s a couple of issues with 0.5.7: (1) free swap is subtracted from total to get the used mem, but total swap is not included (by design) so you end up with a big negative number for mem used. and (2) memory is never changed unless the threshold is exceeded.

    So, in the spirit of cooperation, here is 0.5.8. Changes: (1) configuration items for disable_updates, alert_on_change, and alert_on_error. pretty much self-explanatory. (2) logic bugs above corrected. (3) mem_target is included in the CSV file AND on the graph. (4) sample config file and graphing page is included.

    http://wcgweb.org/memcron-0.5.8.zip

    Hope you find it helpful.

    1. Thanks Bret! Here’s another version that incorporates your changes and drops LWP in favor of using the curl system binary (lowers startup time, and processes the connection faster. I’ll be replacing this again with a bundled version of HTTP::MHTTP when the daemon version is released).

      http://ben.pastebin.com/wmEZgicF

      1. Oh yeah — and memory used is corrected — rather than mem_total, it uses memtotal + swaptotal – memfree – swapfree – cacheused. It *should* be entirely right no matter how your swap is allocated.

      2. I upgraded all to 0.5.9 today even though only 1 machine is giving me the swap oddity. Better safe than sorry I think. Still looking forward to the daemon version when it’s released. Thanks. :)

  21. I wish our memcron memory graph could be updated. I know it doesn’t actually do much itself but seeing more accurate data would put me more at ease.

    Totally off-topic: Anyone using W3 Total Cache? Any thoughts to share?

  22. Kai-

    You can tell if memcron is working properly simply by looking at your daily graph in the DH control panel. Simply look at the most recent value for the red (RAM) line, and as long as your slider is set a little bit higher, you’re good to go. If your slider is set way off, then memcron isn’t working properly.

    Richard

    1. Richard, since upgrading to Ben’s latest version, it doesn’t seem like mine is downsizing. The downsize count (on the busted memcron graph) seems to stay at 1 (I’m set at 3, though my settings don’t make much sense anymore since I can’t figure out anything anymore). For example, used memory on the DH panel graph has been 95-150MB but I’ve been stuck at 1600 (the max on my memcron settings) the entire time. :/

      Thanks for the response though!

        1. Thanks for the updated index.php file. It does look more accurate. At least the available/total memory and used memory seem to match with the DH panel graph. Just need to see if it’ll downsize. It did with the previous 0.5.7(?) version. Hasn’t yet with the 0.5.9 version since I updated it when you posted the link.

          1. Er, I just updated so I won’t know if it works until a bit later but the “win” part was that my feedback helped find a bug.

  23. also, on the config.cfg, you might want to change the setting here to 300 since DH changed their min:

    # Minimum amount of memory that memCron can request (DreamHost default is 300)
    $min_memory = 300;

    (instead of/used to be 150).

  24. Nice work on this. Thanks.

    Here a suggestion for future version. When memory or load spikes I want to know what is causing it. How about the option to have Memcron mail me a dump of “top -c” or some other list of running processes.

    I am not sure if that is even possible via the API, but it would be very helpful.

  25. memCon 0.6 didnt work for me.
    i tried 0.6.2-chris version and it ran, but issued me an error about not finding my PS.
    i thought my PS was called; ps28818

    i have added my API and the PS name into the config file.

    [walsh]$ perl memcron.pl -d
    memcron 0.6.2-chris started (pid: 23378)
    calling api list_ps
    using system curl
    Dreamhost PS walsh not found! at memcron.pl line 138.
    memcron 0.6.2-chris complete

  26. Hi, I just installed memcron and almost sure that I set up everthing as instructed (including chmodding 777 the logs dir) onto http://www.ozar.net/memcron which yields Warning: fopen(/home/ozar/ozar.net/memcron/logs/ps11897.csv) [function.fopen]: failed to open stream: No such file or directory in /home/ozar/ozar.net/memcron/index.php on line 94
    Can’t open log file!

    The emails I receive every 5 minutes say

    “main::normsinv() called too early to check prototype at /home/ozar/ozar.net/memcron/memcron.pl line 94.”

    and they have this subject:
    Cron /usr/local/bin/setlock -n /tmp/cronlock.3196737.93661 sh -c $’perl /home/ozar/ozar.net/memcron/memcron.pl’

    Is this about a bug or misconfiguration?

    Thanks in advance.

  27. When you have replied after 7 years, how do you execpt the reply immediately.Be Patient. Just wait for 7 more years to receive the reply.I use it every day.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>