Version 0.6.4
PsManager is now obsolete as Dreamhost has disabled the API for its VPS service. This happened unexpectedly and with only 2 weeks notice in November 2015.
Dreamhost VPS Manager (short: PsManager) is an open source tool that dynamically manages the size of the memory of your Dreamhost virtual private server.
Managing the size of your memory dynamically, has a certain number of advantages. The main advantage is that you can continue to serve your visitors during peak traffic periods by temporarily increasing available resources without having to pay the monthly fee for these resources, instead you are only charged by Dreamhost for the time the resources were made available to you. Conversely during low traffic periods your resources will be lowered saving you from having to pay for resources that you do not really need all the time. Basically this puts a 4000 MB RAM webserver at your disposal at a fraction of the price.
It also saves you from having to guess how much memory you need and from having to set your server's memory size manually. Dreamhost VPS Manager will do these things for you based on actual current usage.
This program can also notify you, via e-mail or Twitter, of changes to your VPS and reboots of your VPS.
Tip: Not a Dreamhost customer? Save $97 by signing up to Dreamhost with our Dreamhost Promo Codes.
You will need an API key for the Dreamhost web panel.
You can get one here:
Web Panel API.
This API key needs to be allowed to access dreamhost_ps-set_size
and dreamhost_ps-reboot.
It is important for the proper functioning of PsManager that your API key
can access both
these items.
Note: It is also best practice to have a separate API key for each application. Thus even if you already have a API key, do create a new one that will by used only for PsManager.
You can use any shell user on your Dreamhost VPS, but it is probably best to create a new user specifically for PsManager. You can create a new shell user for your Dreamhost VPS here: Manage Users - Add a new user.
Type of User Account: Shell Shell Type: /bin/bash Server: <select appropriate Private Server>
Proceed to download the PsManager script and the configuration file:
psmanager.pl, psrebooter.pl and psmanager.conf.
All three of these files need to be placed in the same directory on your VPS, for example in a "bin" directory of any user, on the private server you want to manage. Avoid placing this script in your cgi-bin directory or under your site's document root, it should not be accessible via your website.
The easiest way do the above is to do it directly from the command line of your VPS. Open an SSH connection to a user on your private server at type the following commands:
mkdir ~/bin 2>/dev/null; cd ~/bin
wget -N http://www.7is7.com/software/dreamhost/download/psmanager.tar.gz
tar -zxvf psmanager.tar.gz
If you did not follow the procedure above, check that the permissions are
in order: psmanager.pl and psrebooter.pl must be executable while
psmanager.conf should only be readable by the owner (or type:
chmod 700 psmanager.pl psrebooter.pl; chmod 600 psmanager.conf).
Note: Is it important that your configuration file is not readable by others as it contains your Dreamhost API key and may contain your Twitter passwords.
When the above is done, edit the configuration file
(psmanager.conf)
filling in your Dreamhost API key.
$key = 'ABCDEFGHIKLM';
Optionally you can have PsManager send notifications of memory size changes and reboots to you via E-mail and/or status updates on Twitter. See "Tweeting and/or Mailing Memory Changes" below for more information about these options.
E-mail setting:
$email_rcpt = 'you@yourdomain.com';
Microblog settings:
$microblog = 'SuperTweet'; # for Twitter $uname = 'my_server'; $pass = 'secret';
Finally to get everything running, you'll need to set up your crontab to run psmanager.pl every minute, add the following line to your crontab (this example assumes the program is located in your "bin" directory):
* * * * * $HOME/bin/psmanager.pl >> /tmp/psmanager_cron.log 2>&1
We recommend to edit your crontab yourself directly on your server, i.e. not to edit it via the DH Control Panel. This is because the Dreamhost control panel adds wrapper code that can disrupt the proper functioning of PsManager.
You can edit the crontab directly with crontab -e, while
crontab -l will list the contents of your crontab.
(Read more about
crontabs
on Dreamhost on their wiki page.)
After collecting a minimum amount of data, PsManager will start to make decisions about what memory size your system needs.
When upgrading from an older version, one generally only needs to
replace psmanager.pl and psrebooter.pl,
as the configuration file can usually
stay the same. You can basically follow the procedure above without replacing
your original psmanager.conf file
The instruction below will only extract the perl programs from the tar-file after downloading it:
cd ~/bin
wget -N http://www.7is7.com/software/dreamhost/download/psmanager.tar.gz
tar -zxvf psmanager.tar.gz psmanager.pl psrebooter.pl
By default Dreamhost servers use Californian time, as this is where they are located. However you may want to see PsManager report changes in an other timezone, like your own. Simply add a line like the one below to your configuration file:
$ENV{'TZ'} = 'Europe/Amsterdam';
To find your timezone, you can either browse through the subdirectories
of /usr/share/timezone/ or use the interactive command line
tool tzselect.
By default Dreamhost Ps Manager will set a memory level between 300 MB and 4000 MB, which are respectively the minimum and maximum allowed by Dreamhost. You can set a higher minimum or lower maximum in the configuration file (the lines are commented out by default).
A sudden increase of popularity of your site (either legitimate or the result of foul play) can cause an increase in allocated memory and higher hosting costs. To avoid the risk of allocating memory beyond what you are prepared to pay for, it is recommended to change the maximum memory setting to a level that you feel comfortable with. An example setting could be:
$mem_max = 2000;
Conversely you can also increase the minimum level of memory. For instance to increase the minimum from 300 MB to 400 MB add this line to the configuration file:
$mem_min = 400;
If the memory changes made are often in a particular range it can help to raise the minimum a bit so as to avoid making downward changes that are very temporary and will not save you much money anyway. Given the limit of 30 changes per day per VPS, you will also be keeping them for more important things.
The buffer setting was introduced in version 0.4.0 to deal with changed memory settings in March 2010. Since that time no extra memory got allocated, instead you had to set the total you wanted. The extra buffer we were given before for free disappeared, but the price of memory was halved at the same time. To compensate for the loss of this buffer, we now maintain a buffer of 50% by default to preserve burstability of server. However this can be changed; lower it if you want to live dangerously and risk running out of memory or raise it if you want more peace of mind, though we think 50% (the default) is about right.
$buffer_perc = 50; # Memory buffer in percent
In version 0.6.2 we have added the possibility to set a minimum buffer size in MB. By default this is now set to 200, but can be set to anything else in the configuration file. (Note that less than 200 MB probably doesn't make too much sense.)
$buffer_min = 500; # Minimum buffer size (in MB)
You can also create more (or less) space by applying a factor to the memory calculation other than 1 (which is the default). For example if you want to add 20% to the calculated memory, set memory_factor to 1.2.
$memory_factor = 1.2;
If you want to reserve more memory for caching purposes you could consider to increase the allocated memory by a more significant factor. This is not needed for the proper functioning of your webserver but it can't hurt either and will speed up your server (and sites) by significantly speeding up disk access (both write and read).
To increase the allocated memory by half:
$memory_factor = 1.5;
In version 0.6.2 the option to set a minimum extra memory (in MB)
was introduced.
If the $memory_factor is equal to or greater than 1
PsManager will add the number that is the highest. For instance if the
memory factor is 1.2 and the minimum extra is 200 MB, it will add 200 MB to
any total below 1000 but add more than 200 to totals above 1000.
By default this is set to 0.
$memory_minextra = 100; # Minimum extra memory (in MB)
If $memory_factor is less than 1, the setting of
$memory_minextra functions like a treshold.
PsManager will only reduce the memory by the amount that exceeds the minimum
extra memory setting.
For instance if your settings are to shave off 10% with a 100 MB threshold, PsManager will not shave off anything until the total memory exceeds 1000 MB (as 100 MB = 10% of 1000 MB), above 1000 MB it will shave off 10% minus 100 MB. For example when the total is 1500 MB, 10% minus 100 MB is 50 MB, so the total will be set to 1450 MB instead of 1500 MB.
$memory_factor = 0.9; $memory_minextra = 100; # Functions as a threshold
While it doesn't really matter if you use the buffer setting or the memory factor to add more memory, the idea behind the two settings is to deal with two separate aspects: burstability and caching. Cache memory is mainly used to speed up disk access (often accessed files are kept in memory). Cached memory is always available for processes when needed, and so it is in reality freely available memory that could also be used for burstability purposes if required and vice-versa memory for burstability will be used for caching when it's not in use.
Note: $buffer_perc and $memory_factor
are applied sequentially as follows:
first the actual memory used by processes is increased with an amount
sufficient to maintain a buffer of the specified percentage. That result
in turn is multiplied by $memory_factor to give us the amount of memory
we want to reserve.
At this time, dreamhost is unable to resize a VPS to a level below its actual usage plus cache. While cache is free memory that is available at all times for processes, for some reason when downsizing a VPS this can not be freed up.
The only option left to us is to force a reboot, which will clear the cache and allow the resize to be made. We have introduced a limit to control when to force a reboot. With $force_lim set to 20 for instance, psmanager is instructed to force a reboot if the amount of memory reserved would be more than 20% higher than what is optimal.
$force_lim = 20;
Although reboots are fast they still cause a bit of downtime, hence you can avoid reboots when you think they are not worth the amount of memory saved.
To prevent PsManager from rebooting your VPS, simply increase the limit to a very high percentage that will never be reached.
$force_lim = 1000;
Note: We are hoping for Dreamhost to provide us with an API call to clear the cache as that would remove the need for reboots. A more detailed explanation of this problem can be found on the Dreamhost forum, in the thread entitled: "We had to reboot your VPS".
PsManager should be able to determine the server you want to manage automatically. If for some reason it does not, you can manually set the 'ps' variable in the configuration file to indicate the correct machine.
$ps = 'ps0000';
As of version 0.2.4 you can set complex Perl instructions in the configuration file, for example to increase the minimum required memory at certain times of the day. For example the code below will set the minimum memory to 400MB from 7pm (19:00) to midnight.
{
my ($sec,$min,$hour) = localtime();
if ($hour>=19 && $hour<=23) {
$mem_min = 400;
$memory_factor = 1;
}
}
You can have PsManager send you e-mails and/or status updates about memory changes and reboots.
A really cool feature (or at least it was cool when Twitter was still cool) is undoubtedly that you can follow changes to your Dreamhost VPS via Twitter. The idea is simple, you create a Twitter account for your Dreamhost VPS and let PsManager update it. You then proceed to follow this account from your own Twitter account. Each time the memory of your Dreamhost VPS changes you'll see a tweet about the change appear in your updates stream.
Note: As Twitter has closed down Basic Authorization for its API, direct Twitter updates no longer work. However Twitter can still be updated in following way:
Unfortunately Twitter's new OAuth mechanism isn't very user-friendly (going via SuperTweet is a lot easier for anyone to set up). OAuth would take a significant effort to implement, and would add a lot of bloat to our software. We are hoping Twitter will offer a more user-friendly solution in the not too distant future, something as easy as SuperTweet.
You can check how PsManager tweets the changes it makes to my Dreamhost VPS:
It is possible to define your own template for the status update e-mails and tweets, by adding a line to the configuration file (the {tz} tag was added in version 0.5.0). Template example for memory size updates:
$tweet_template = "{ps}: {change} {oldsize}MB → {newsize}MB at {time} [{tz}]";
Template example for detected reboots:
$reboot_template = "{ps}: rebooted at {time} [{tz}]";
Where:
{ps} = the name of your VPS
{newsize} = the new memory allocation in MB
{oldsize} = the old memory allocation in MB
{time} = the time at which this change occurred
{tz} = the timezone offset
{change} = A visual representation of the change with - and + signs, with
the number of signs rounded up to the next multiple of 100 MB.
for instance: +++ = increase of between 200 and 300 MB
Since version 0.6.1, it is possible to run PsManager in monitor only mode.
It will log actual memory usage, changes and reboots but not request any
changes itself. Just set $monitor_only to 1 in the configuration
file.
$monitor_only = 1;
If you are encountering problems, these are a few starting points to check:
Check that the cronjob is set properly and runs every minute:
crontab -l | grep psmanager
Check the output file for the PsManager cronjob. It should be empty if there are no problems. Errors will be sent here:
tail /tmp/psmanager_cron.log
Check the usage log maintained by PsManager:
tail ~/bin/psmanager.log
The columns are: <timestamp> <available memory> <used memory> <free memory> <cache memory> <optimal memory> <set memory> <event>. All memory in MB.
Optimal memory is the amount of memory PsManager
thinks you should have at this very moment.
The set memory column indicates that PsManager gave the
instruction to change the memory to the given level.
The log extract below shows how memory is changed from 400MB to 316MB, 316 was calculated to be the optimal size at that moment. At one point a new size is set (S), the next minute the change (C) has taken effect:
201011250757 400 43 356 229 302 0 201011250758 400 43 356 229 298 0 201011250759 400 41 358 230 294 0 201011250800 400 46 353 230 316 316 S 201011250801 316 46 269 231 311 0 C 201011250802 316 46 269 231 304 0 201011250803 316 31 284 232 297 0
Our second example extract shows how a sudden increase in usage (from 52MB to 165MB) and a drop in the cache from 117 MB to 55 MB:
201011251249 300 54 245 116 276 0 201011251250 300 52 247 117 273 0 201011251251 300 165 134 55 286 0 201011251252 300 66 233 209 294 0 201011251253 300 54 245 236 293 0
List all events:
grep -e '[BCS]' ~/bin/psmanager.log
Events are: B = reBoot detected, C = memory Change detected, S = memory Set.
Verbose logging (implemented since version 0.5.0)
will keep a log in ~/bin/psmanager_verbose.log
with events and details of each run. This is useful to look for possible
problems or to keep a detailed log for future reference.
To turn on verbose logging add this line to the configuration file:
$verbose = 1;
To list the latest (40) entries in the log simply use tail:
tail -40 ~/bin/psmanager_verbose.logTo find recent error messages:
tail -1000 ~/bin/psmanager_verbose.log |grep " E:"
Each line in the verbose log has an ID character. They are as follows: A = Api, B = (re)Boot, D = Debug, E = Error, I = Info, L = Logfile, M = Mail, R = Report.
An Info line looks like this: after the timestamp and I flag, we see the main settings, minimum and maximum memory, buffer settings, extra memory settings and the setting of the monitor only flag.
201208091700 I: Min:800 Max:4000 Buf:50|500 Ext:1.1|200 Mon:0
A Report line contains the same first 4 numbers that appear in the normal log file, followed by normal usage over the past hour and a correction for the trend. Between brackets is the per minute trend that was based on. We then have the optimal memory after applying all the buffers and extras followed by any events (or 0 if none). The last 4 numbers are different counters: number of resizes in the last 8 hours, minutes since a trigger occurred, minutes since last change and minutes since last reboot (479 means no line in the log had a reboot).
201208091659 R: 1065/278/786/785 389.8-49.6(-1.65) =1040|0 [3|0|146|479]
Note: Since version 0.6.4 the verbose log is regularly trimmed to contain about 1 month worth of log lines. If you are using an older version, you may want to clear out old verbose logs every once in a while, or upgrade to the newest version.
If you are encountering problems, and need help, PsManager can generate a report, that you can copy and paste into an e-mail. This checks a number of settings and the things mentioned above, while not disclosing any sensitive information like account names, passwords and API keys. To generate this report run psmanager.pl from the command line with the "check" argument:
~/bin/psmanager.pl check
PsManager monitors memory usage over the last 60 minutes and calculates an optimal memory requirement from that. Technically it takes the average of the 8 highest memory usage peaks (discarding the two highest ones) in the last 60 minutes. Single peaks into the memory buffer are acceptable, but regular use of the memory buffer is not advisable as it increases the risk of running out of memory all together. This is why we make an effort to avoid depending on the memory buffer and keep reserved memory on the high end of actual usage.
Dreamhost limits us to making 30 changes per VPS per day to the memory size, as a result we have to be a bit careful with changes. To avoid running into this restriction PsManager will never make more than 10 changes in 8 hours. In order to avoid making too many up and down changes, downward changes will occur only if no changes have been made in the previous 60 minutes and the change exceeds 10% of current reserved memory. If more than 4 changes have already been made in an 8 hour period, we are even more careful and only downsize after 120 minutes. As a result in any 4 hour period, a minimum of 3 changes are always reserved for upward corrections.
As it is more important to maintain availability of your sites a change will be made upwards if 30 minutes have passed and it exceeds 10%. If the number of changes in the past 8 hours exceeds 4 we will instead wait for 1 hour to pass. Unless the required increase exceeds 20% after 10 minutes (indicating a possible surge in traffic to your sites).
PsManager will also never use the last two available resizes in a 8 hour period for a 10% change, but instead reserve it for an emergency upsizing of at least 20%, as availability of your site is of paramount importance.
Version 0.2.6 introduced the usage trend in the calculation, if memory usage is going up over an hour it will reserve a bit more than current usage would indicate, and conversely if it is going down it will reserve a bit less. Since 0.5.0. this variation can fluctuate between -15% and +15%.
Versions 0.3.0 and 0.4.0 deal with the continuing changes Dreamhost made to the memory setup of new virtual private servers. Thankfully those changes are now well behind, the new Dreamhost setup is much better for efficient management of memory allocation.
Version 0.4.3 removes the list_ps API calls, making
it possible to manage multiple VPS servers on the same account without
hitting the API rate limit. This version now also tweets actual memory
updates instead of requested updates. Actual updates can lag the request
and sometimes happen in several steps - that is now visible.
Since version 0.5.0 PsManager keeps the log for 8 hours, making it possible to change reserved memory faster if few changes are made over an 8 hour period. The log file now also keeps track of memory used for disk caching. Version 0.5.0 also added the timezone tag, verbose logging and the self check report. It also fixed a few minor bugs.
Version 0.5.1 introduced a new way to update Twitter (via SuperTweet), and version 0.5.2 to 0.5.4 fixed some minor bugs, changed some message layouts and optimized some code.
Version 0.5.5 fixes some minor bugs and includes a workaround for the problem of not being able to resize below the level of memory and cache. Hopefully that problem will be fixed by Dreamhost, at which time this workaround will be disabled. Version 0.5.6 adds the possibility to reboot a VPS if cache usage makes it impossible to resize to the optimal memory level.
The most visible new features in version 0.6.0 are the possibility to e-mail status updates and system reboot detection. Besides that it includes several improvements to help troubleshooting. In version 0.6.1 we introduced the monitoring only flag.
Version 0.6.2 replaced the previous crude trend calculation with one based on simple linear regression. It also introduced the option to set minimum buffer memory and minimum extra memory (instead of only being able to set a percentage or factor).
As an active user of the tool myself, I am constantly monitoring how it performs and occasionally make tweaks or add features. Suggestions based on your own experience are of course welcome. Some suggestions have led to new features being integrated (for example the templated tweets with the change visualization with + and - signs). Others have helped a lot with testing on their very active virtual servers allowing me to fine tune the program to better deal with those situations.
Guaranteed freedom of use and full transparency of code.
The PsManager program is released under an open source license (GNU-GPL v3), which means that you can use this script for free and make any modifications that you please. You are in no way dependent on the original author of this program for future maintenance and your right to use it never expires. The source code of the script is easily viewable (the script is written in perl), you can scrutinize it or have it scrutinized to convince yourself that it does what it says it does.
If you are using the Dreamhost VPS Manager script (PsManager), you may also be interested in using the Dreamhost VPS Rebooter script (PsRebooter) to automatically schedule reboots (which clears out any hanging processes).
If you are having trouble with sudden unexplained peaks of memory consumption, Memlog might be of interest to you.
There is a Dreamhost forum thread for this application where you can ask questions. If you are using or want to use PsManager feel free to contact me by E-mail about any problems or suggestions. If you don't want to do the installation of PsManager yourself, I can also do the whole installation for you on your VPS in exchange for a small fee.
Many features are the result of feedback from people who told me what they would like to be able to do, or what problems they encountered. Several bugs have been fixed thanks to user feedback. Feedback has been valuable to make PsManager what it is today, therefor a special thanks to everybody who has sent feedback.
Note: If you want to be notified of important new releases of PsManager, or problems affecting PsManager, send me an E-mail and ask to be added to the mailing list. (It is still a small mailing list that I maintain manually.)
7is7.com | Software | Otto | Travel Stories | Countdown Clock | Firefox | StatEye
New | About | Contact | Connect | Friends | Promotions | Copyright | Advertise
