patch

Ubuntu Security Updates

Question: 

How do I apply Ubuntu security updates to my Stackato cluster?

Answer: 

Ubuntu Security Updates

Both the Stackato VM and the Docker base image used for application containers run Ubuntu. To maintain an up-to-date system with all known security patches in place, the VM and Docker base images should be updated with the following process whenever an important security update becomes available in the Ubuntu repositories.

Upgrade the Stackato VM

Run the following commands on all cluster nodes, one node at a time:

sudo apt-get update
sudo unattended-upgrades -d

If you are using a proxy you may need to export http_proxy and https_proxy environment variables. For example:

sudo sh -c "http_proxy=http://myproxy.example.com:3128 https_proxy=http://myproxy.example.com:3128 unattended-upgrades -d"

This will run the unattended-upgrades script to install all upgrades from the "-security" stream.

Each node should be rebooted after unattended-upgrades completes to ensure new kernels, modules, and libraries are loaded.

2.10 - Patch the LXC Container Template

Follow the process laid out in this previous FAQ:
http://community.activestate.com/node/10406

3.X - Upgrade the Docker image

The base Docker image used for application containers should also be upgraded once the VM is up-to-date. Perform the following steps on each DEA node in the cluster, one node at a time:

Create a new working directory:

mkdir ~/upgrade-alsek && cd $_

Create a Dockerfile. In this new directory, create a file called "Dockerfile" and add the following:

FROM stackato/stack-alsek
RUN apt-get update
RUN unattended-upgrades -d
RUN apt-get clean && apt-get autoremove

Build the docker image. Give the image a tag relevant to this particular upgrade (e.g. "upgrade-2014-09-19"):

sudo docker build -rm -t stackato/stack-alsek:upgrade-2014-09-19 .

The "." at the end is important. It specifies to use the Dockerfile in the current directory.

Tag the Docker image as the "latest" stack-alsek image:

sudo docker tag stackato/stack-alsek:upgrade-2014-09-19 stackato/stack-alsek:latest

All running applications will need to be restarted by their owners or Stackato admins (using the management console or the stackato client) in order for security upgrades to take effect within their application containers. You can check which image running apps are using by running docker ps on your DEAs (but do not use docker restart).

If you have DEA autoscaling enabled, be sure to also update the DEA template

For more information, see the Stackato documentation:
docs.stackato.com/admin/best-practices/index.html#ubuntu-security-updates

Contents of Batch Patch-05082014 - 3.4.1

Question: 

What fixes were included in the 3.4.1 batch patch-05082014?

Answer: 

This patch included the following fixes:

-Fixed the way we read the autoscaling.yml file (/s/etc/autoscaling/autoscaling.yml) to allow us to properly set the security group(s) for autoscaled DEAs in EC2.
-Fixed cloud controller to properly allow deletion of applications for which the controller database has services which don't exist. The controller will now properly delete these applications rather than throwing an error.

Stackato 3.4 Patch: external-postgres-9.3

Question: 

What's in patch external-postgres-9.3?

Answer: 

external-postgres-9.3 allows the PostgreSQL data service to be compatible with an external PostgreSQL 9.3 instance. It was previously only compatible with PostgreSQL 9.1. This patch will restart PostgreSQL data service.

For more information on setting up an external data source, please see the Stackato documentation: http://docs.stackato.com/admin/cluster/external-db.html#postgresql

Kato patch external hosts

Question: 

I need to manually whitelist hosts in my corporate proxy. What hosts does kato patch need to access?

Answer: 

The hosts that kato patch needs to access may vary with local configuration, and may change depending on changes to 3rd party services such as rubygems.org.

We recently audited kato-patch and found it accesses the following hosts:

http://archive.ubuntu.com
http://get.docker.io
http://mirrors.kernel.org
http://security.ubuntu.com
https://a248.e.akamai.net
https://bb-m.rubygems.org
https://bundler.rubygems.org
https://get.stackato.com
https://github.com
https://registry.npmjs.org
https://rubygems.org
https://s3.amazonaws.com

Heartbleed vulnerability and Stackato

Question: 

Is Stackato vulnerable to the Heartbleed bug? How can I patch Stackato?

Answer: 

Stackato is vulnerable to the Heartbleed bug. You can patch your system by running "kato patch install heartbleed-fix". This patch will install updated OpenSSL libraries on both the host VM and inside the container templates. Most apps won't need to be redeployed, but some may depending on the app and how SSL is used. We advise you test your app (see tools below) after applying the patch to determine if redeploying the app is necessary.

A patch is also available for Stackato 2.10.4 here:
https://get.stackato.com/patch/2.10/stackato-2.10.4-heartbleed.sh

If you have any questions please contact ActiveState support.

More info:

http://heartbleed.com/
http://filippo.io/Heartbleed/
https://gist.github.com/sh1n0b1/10100394
https://github.com/titanous/heartbleeder

Stackato 2.10.6 Ruby Security Patch

Question: 

I notice that ruby released a critical security update recently. Is stackato impacted by this?

Answer: 

Stackato is indeed affected by this. A patch has been generated and is available via 'kato patch'. As always this can be installed via 'kato patch status' followed by 'kato patch install.

Notes: This patch downloads a 50 MB tar file from our public download site, and will do so on every node in your cluster. This file will be removed once the patch is installed.

Additionally, this patch will restart EVERY role on EVERY node on your cluster as stackato makes significant use of ruby in a number of places. The outage shouldn't last more than a minute for each node though this might vary slightly depending on your IaaS solution.

Stackato 2.10.6 - Patching Linux on a Stackato VM

Question: 

How do I ensure that my Stackato nodes are running the latest patches released for Ubuntu?

Answer: 

Stackato 2.10.6 supports the use of the standard Ubuntu "apt-get" patching tools. Critical Stackato components are protected by pinning the affected packages, so users are able to run
'sudo apt-get update&&sudo apt-get dist-upgrade'
from a terminal window (ssh) to patch the base VM. Each VM will need to be patched separately.

Pinning packages means that apt-get is unlikely to result in a reboot being needed, however, it is still good practice to check the content of the recommended patches. Planning for the use of maintenance mode, with a contingency of a controlled reboot if appropriate, is strongly recommended.

Pinned packages can only be updated by upgrading to a newer version of Stackato. To see a list of pinned packages, execute 'sudo apt-mark showhold'.

After patching the base VM, the template for generating new LXC containers should also be patched. See the FAQ for patching the templates:
http://community.activestate.com/node/10406

While it is possible to 'stackato ssh' into a running container and patch it if you have sudo enabled for that container, we do not recommend this. Patching the template, spinning up new containers, and dropping the old container will give more consistent results over the long term.

Stackato 2.10.X NodeJS security fix v0.10.21

Question: 

I saw that NodeJS recently released a critical security patch. Is Stackato effected by this as it uses nodejs for the router component?

Answer: 

Yes, this vulnerability does impact Stackato. We have already generated a patch which will replace the existing Node version with their updated version.

###2.10.4
You can download the patch at http://get.stackato.com/patch/2.10/stackato-2.10.4-nodejs-security-fix.sh. This patch will need to be applied to any nodes running the 'router' role as well as your core role (which acts as a router). The patch can be applied via 'sh stackato-2.10.4-nodejs-security-fix.sh'. After applying this patch you should restart your router role by executing 'kato restart router'.

###2.10.6
This patch is available for 2.10.6 through the kato patch command. You will need to update your manifest via 'kato patch status', and you can install the patch after doing that by executing 'kato patch install'. This patch will require a sudo password, and you should note that it will restart your router.

Stackato 2.10.X Security Fix - container sudo fix

Question: 

Any recent security patches for Stackato?

Answer: 

We've generated a second patch that needs to be applied on top of the initial apt-get-wrapper patch to fix an issue that this was causing with unprivileged users in containers.

###2.10.4

First step is to install everything at http://community.activestate.com/node/10157. This will include http://get.stackato.com/patch/2.10/stackato-2.10.4-apt-get-wrapper.sh, which has an issue as described above. This can be corrected with the patch downloaded at http://get.stackato.com/patch/2.10/stackato-2.10.4-apt-get-wrapper-fix.sh. Patch instructions are to upload the patch to all nodes running 'DEA' or 'Stager' in your cluster and execute via 'sh stackato-2.10.4-apt-get-wrapper-fix.sh'. Upon doing this you should restart your stager and/or dea roles. Any future applications deployed will have this fix enabled.

###2.10.6

This patch is available via the kato patch command and can be installed by executing 'kato patch update' followed by 'kato patch install'.

Stackato 2.10.6 - Patching the LXC container template

Question: 

When Ubuntu patches are issued, how do I apply those updates to the template used to create new LXC containers?

Answer: 

We have a script to assist with this.

http://get.stackato.com/patch/upgrade-container.tar.gz

The tarfile contains the script and Installation instructions. It will need to be run on each node running a DEA or a Stager at any time that you would like to apply updates*. This should be done interactively from the command line, so that you can monitor the progress.

New containers launched after our script is run will have the updates applied. Containers started before the updates were applied will not be altered, but you can migrate running droplets to updated containers on other DEAs. See this FAQ for a more detailed description of moving apps.
http://community.activestate.com/node/10219

*Note that the actual updates that will be installed are provided by the Ubuntu community, and what you install when you run our script will depend on what is available, and what you have already installed.