How do I apply Ubuntu security updates to my Stackato cluster?
Both the Stackato VM and the Docker base image used for application containers run Ubuntu. To maintain an up-to-date system with all known security patches in place, the VM and Docker base images should be updated with the following process whenever an important security update becomes available in the Ubuntu repositories.
Run the following commands on all cluster nodes, one node at a time:
If you are using a proxy you may need to export http_proxy and https_proxy environment variables. For example:
sudo sh -c "http_proxy=http://myproxy.example.com:3128 https_proxy=http://myproxy.example.com:3128 unattended-upgrades -d"
This will run the unattended-upgrades script to install all upgrades from the "-security" stream.
Each node should be rebooted after unattended-upgrades completes to ensure new kernels, modules, and libraries are loaded.
Follow the process laid out in this previous FAQ:
The base Docker image used for application containers should also be upgraded once the VM is up-to-date. Perform the following steps on each DEA node in the cluster, one node at a time:
Create a new working directory:
mkdir ~/upgrade-alsek && cd $_
Create a Dockerfile. In this new directory, create a file called "Dockerfile" and add the following:
Build the docker image. Give the image a tag relevant to this particular upgrade (e.g. "upgrade-2014-09-19"):
sudo docker build -rm -t stackato/stack-alsek:upgrade-2014-09-19 .
The "." at the end is important. It specifies to use the Dockerfile in the current directory.
Tag the Docker image as the "latest" stack-alsek image:
sudo docker tag stackato/stack-alsek:upgrade-2014-09-19 stackato/stack-alsek:latest
All running applications will need to be restarted by their owners or Stackato admins (using the management console or the stackato client) in order for security upgrades to take effect within their application containers. You can check which image running apps are using by running docker ps on your DEAs (but do not use docker restart).
If you have DEA autoscaling enabled, be sure to also update the DEA template
For more information, see the Stackato documentation:
What fixes were included in the 3.4.1 batch patch-05082014?
This patch included the following fixes:
-Fixed the way we read the autoscaling.yml file (/s/etc/autoscaling/autoscaling.yml) to allow us to properly set the security group(s) for autoscaled DEAs in EC2.
-Fixed cloud controller to properly allow deletion of applications for which the controller database has services which don't exist. The controller will now properly delete these applications rather than throwing an error.
What's in patch external-postgres-9.3?
external-postgres-9.3 allows the PostgreSQL data service to be compatible with an external PostgreSQL 9.3 instance. It was previously only compatible with PostgreSQL 9.1. This patch will restart PostgreSQL data service.
For more information on setting up an external data source, please see the Stackato documentation: http://docs.stackato.com/admin/cluster/external-db.html#postgresql
I need to manually whitelist hosts in my corporate proxy. What hosts does kato patch need to access?
The hosts that kato patch needs to access may vary with local configuration, and may change depending on changes to 3rd party services such as rubygems.org.
We recently audited kato-patch and found it accesses the following hosts:
Is Stackato vulnerable to the Heartbleed bug? How can I patch Stackato?
Stackato is vulnerable to the Heartbleed bug. You can patch your system by running "kato patch install heartbleed-fix". This patch will install updated OpenSSL libraries on both the host VM and inside the container templates. Most apps won't need to be redeployed, but some may depending on the app and how SSL is used. We advise you test your app (see tools below) after applying the patch to determine if redeploying the app is necessary.
A patch is also available for Stackato 2.10.4 here:
If you have any questions please contact ActiveState support.
I notice that ruby released a critical security update recently. Is stackato impacted by this?
Stackato is indeed affected by this. A patch has been generated and is available via 'kato patch'. As always this can be installed via 'kato patch status' followed by 'kato patch install.
Notes: This patch downloads a 50 MB tar file from our public download site, and will do so on every node in your cluster. This file will be removed once the patch is installed.
Additionally, this patch will restart EVERY role on EVERY node on your cluster as stackato makes significant use of ruby in a number of places. The outage shouldn't last more than a minute for each node though this might vary slightly depending on your IaaS solution.
How do I ensure that my Stackato nodes are running the latest patches released for Ubuntu?
Stackato 2.10.6 supports the use of the standard Ubuntu "apt-get" patching tools. Critical Stackato components are protected by pinning the affected packages, so users are able to run
'sudo apt-get update&&sudo apt-get dist-upgrade'
from a terminal window (ssh) to patch the base VM. Each VM will need to be patched separately.
Pinning packages means that apt-get is unlikely to result in a reboot being needed, however, it is still good practice to check the content of the recommended patches. Planning for the use of maintenance mode, with a contingency of a controlled reboot if appropriate, is strongly recommended.
Pinned packages can only be updated by upgrading to a newer version of Stackato. To see a list of pinned packages, execute 'sudo apt-mark showhold'.
After patching the base VM, the template for generating new LXC containers should also be patched. See the FAQ for patching the templates:
While it is possible to 'stackato ssh' into a running container and patch it if you have sudo enabled for that container, we do not recommend this. Patching the template, spinning up new containers, and dropping the old container will give more consistent results over the long term.
I saw that NodeJS recently released a critical security patch. Is Stackato effected by this as it uses nodejs for the router component?
Yes, this vulnerability does impact Stackato. We have already generated a patch which will replace the existing Node version with their updated version.
You can download the patch at http://get.stackato.com/patch/2.10/stackato-2.10.4-nodejs-security-fix.sh. This patch will need to be applied to any nodes running the 'router' role as well as your core role (which acts as a router). The patch can be applied via 'sh stackato-2.10.4-nodejs-security-fix.sh'. After applying this patch you should restart your router role by executing 'kato restart router'.
This patch is available for 2.10.6 through the kato patch command. You will need to update your manifest via 'kato patch status', and you can install the patch after doing that by executing 'kato patch install'. This patch will require a sudo password, and you should note that it will restart your router.
Any recent security patches for Stackato?
We've generated a second patch that needs to be applied on top of the initial apt-get-wrapper patch to fix an issue that this was causing with unprivileged users in containers.
First step is to install everything at http://community.activestate.com/node/10157. This will include http://get.stackato.com/patch/2.10/stackato-2.10.4-apt-get-wrapper.sh, which has an issue as described above. This can be corrected with the patch downloaded at http://get.stackato.com/patch/2.10/stackato-2.10.4-apt-get-wrapper-fix.sh. Patch instructions are to upload the patch to all nodes running 'DEA' or 'Stager' in your cluster and execute via 'sh stackato-2.10.4-apt-get-wrapper-fix.sh'. Upon doing this you should restart your stager and/or dea roles. Any future applications deployed will have this fix enabled.
This patch is available via the kato patch command and can be installed by executing 'kato patch update' followed by 'kato patch install'.
When Ubuntu patches are issued, how do I apply those updates to the template used to create new LXC containers?
We have a script to assist with this.
The tarfile contains the script and Installation instructions. It will need to be run on each node running a DEA or a Stager at any time that you would like to apply updates*. This should be done interactively from the command line, so that you can monitor the progress.
New containers launched after our script is run will have the updates applied. Containers started before the updates were applied will not be altered, but you can migrate running droplets to updated containers on other DEAs. See this FAQ for a more detailed description of moving apps.
*Note that the actual updates that will be installed are provided by the Ubuntu community, and what you install when you run our script will depend on what is available, and what you have already installed.