Wednesday, May 18, 2016

Acer Aspire Ubuntu "Freezing" Issue (Ubuntu 14.04, 16.04).

Starting from late last year, and continuing into the time of writing (May 2016), there have been several long threads on the kernel bug tracking system (bugzilla.kernel.org), the Ubuntu Forums and Launchpad with dozens of infuriated Acer Aspire users experiencing spontaneous, randomly-timed freezes which affect Ubuntu 14.04 (Trusty Tahr), Ubuntu 16.04 (Xenial Xerus), and perhaps earlier releases too (although I cannot find any evidence for this). 

One user tried downgrading his kernel; another upgrading his kernel; both to no avail.  Later another user reports it to be a "...horrible bug affecting BayTrail users", with "little activity from Intel or kernel devs to fix it".  Bay Trail is an an SoC architecture common on many Acer Aspire machines. 

The good news is that there's one solution which truly works; the bad news is that it will severely affect your battery life.  So don't chuck out your laptop, but remember to put that power supply into your bag before going to Uni or work.

Here's the fix.  Add the following parameter into Grub:


intel_idle.max_cstate=1

When altering kernel parameters you should always 'test' it out first before applying it permanently.

To 'test' out the kernel parameter enter into grub (press "SHIFT" repeatedly during your computer's POST (power-on-self-test) phase) to enter grub and you should enter a GRUB screen which looks something like this:

It might look different depending on your desktop environment.  I use Ubuntu Mate and mine looks like this:

As you can see the options are still the same; they just look different.

Select the option which your normally boot from and press 'e' to edit it.  Locate the line mentioned earlier beginning with 'linux', and add the new bootarg  intel_idle.max_cstate=1
(sometimes also called a "boot parameter" or bootparam) somewhere towards the end of the line.  Press ctrl+x or F10 to boot using this param.  If the kernel can boot successfully without any issues and you can login, you know this fix is safe to apply for good and for every time you reboot your system.

Type the following command into your terminal:
sudo vim /boot/grub/grub.cfg

Make the same modification that you made from the Grub screen, save and exit.  You will most likely need to save with "w!" because grub.cfg is a read-only file.

Now this bootarg will be set for the next time you reboot and every subsequent reboot; and hopefully for you, the freezing issue will be gone.  But at what cost?  We mentioned earlier that there was a power-consumption side effect to this fix.  It has to do with a CPUs "C-states" and "P-states".

"C-states" are "power consumption states" and can be thought of as different levels of "sleep".  A CPU, or any one of its individual cores, if it is not currently doing anything would effectively be "driving around in circles", which wastes power.  The solution to this is that CPUs have different "C-states" starting from C0 right up to Cn where n is the highest C-state.  The higher the C-state level and the higher the value of n, the lower the transistors' quiescent (resting) voltage is, and the lower the transistors' switching frequency is; both of which drastically affect power consumption.  CPUs enter high levels of "n", during longer periods of "doing nothing".

Looking back at the fix we applied, we have limited our Baytrail chip's C-state to "1" (intel_idle.max_cstate=1).  One could only deduct from this fix that entering into one of the higher C-states or exiting out of them is what is causing the kernel to freeze.

So-called "P-states" are to do with performance.  During periods where the CPU is executing tasks, but the CPU utilization is not 100%, the CPU may enter into high "P-states"  in order to further save power.

Saturday, January 2, 2016

Logwatch for a Ubuntu 14.04 public-facing SSH sever

The following post will show you how to set up Logwatch on a Ubuntu machine with a public facing SSH server, so that every day you are e-mailed a daily summary of stats, as per the screenshot below.  In a separate blog post, I will include a very easy mechanism to block the offending IP addresses of repeated failed logins.  This might all sound very daunting, but it is in fact, very easy to set up.  So without further ado, let's get started.

This post assumes that you are dealing with a public-facing SSH server.  That is, a small or home office Ubuntu machine which you are able to ssh into from the outside world.

This post will also assume you are using Ubuntu 14.04.x LTS.  The advice contained within this post may also work for other Ubuntu versions.

If you are not already running an SSH server on your Ubuntu machine, it's very easy to get it up and running;

sudo apt-get install openssh-server

After it finishes installing, your system will automatically get it up and running (without rebooting).  You can check for the existence of a running server with a simple grep.

ps aux | grep sshd
root      6958  0.1  0.1  61372  5048 ?  Ss   15:02   0:00 /usr/sbin/sshd -D

Note: The rest of this post assumes you can ssh into your machine from the outside world.  For example, you should be able to ssh into your machine using an app your mobile device.  If you can't, then there's something blocking your Ubuntu machine from being accessible to the outside world, and you'll need to address that problem before continuing with the rest of this post.  If your Ubuntu machine resides at home behind an ISP-provided router, you will most likely need to research what port forwarding is, and learn how to set that up on your home router.

Your SSH server's configuration file (config file) lives at /etc/ssh/sshd_config.  There are various options you can set here such as changing the listening port, allowing or disallowing root logins, and so on.  You can look at man sshd_config or google for "sshd_config options" for more info.  Every time you edit this file you will need to restart your SSH server.

sudo service ssh restart

RSYSLOG

rsyslog is the system logger which comes with Ubuntu by default.  You'll want to modify its configuration file so that it redirects all messages related to your SSH server to a separate log file.

Add the following line (anywhere) to your /etc/rsyslog.d/50-default.conf:

# Redirect sshd messgaes
if $programname == 'sshd' then /var/log/sshd/sshd.log

Every line beginning with a hash is a comment.  After modifying the file, restart the service.

sudo service rsyslog restart

After that, you'll notice that every time you try to remotely log in to your Ubuntu machine via ssh, bungle the username or password, disconnect from the server, start, stop or restart the sshd service, some diagnostic messages will appear in /var/log/sshd/sshd.log.

For a while, these messages might seem informative and and useful, but over time the messages will grow out of control and be too cumbersome to manage and analyze.  This is where logwatch comes in handy.

 sudo apt-get install logwatch

When it comes up with a graphical prompt, choose "Internet Site", as we will later be using SMTP.  Change the System Mail Name to any name of your choosing.  You can choose a name which describes your machine.  Use your favourite text editor to create the file /etc/logwatch/conf/logfiles/sshd.conf.  This file only needs to contain one line:

Logfile = /var/log/sshd/sshd.conf

Now open the following file with your editor:
/usr/share/logwatch/default.conf/logwatch.conf

In that file you'll find a section starting with "Service = All".  Comment out all the "Service =" lines (including Service = All) and add in your own line at the bottom, just for our ssh server;

Service = "sshd"

Also change the line #mailer = "/usr/sbin/sendmail -t" to mailer = "/usr/sbin/sendmail" (we will install this program in the next section).

The range parameter should be set to to 'today', the MailTo should be the name of your email address, and and MailFrom can be anything.

As you might recall from the beginning of the post, logwatch e-mails you an easy-to-digest summary of the logfiles on your system which your system logger produces.  For this, logwatch is going to need a way to send out emails.  The next section details this.  For reasons which will become apparent soon, you will need to set yourself up a fresh email account (in this post I will be using a Gmail account).

SSMTP
Proceed to install an SMTP (Simple Mail Transfer Protocol) client on your machine;

 sudo apt-get install ssmtp

Now edit the configuration file, /etc/ssmtp/ssmtp.conf.  Delete all the lines and start over, like so:
root=YOUR_EMAIL_ADDRESS
mailhub=smtp.gmail.com:587
rewriteDomain=YOUR_EMAIL_DOMAIN
AuthUser=YOUR_USERNAME
AuthPass=YOUR_PASSWORD
FromLineOverride=YES
UseTLS=YES
UseSTARTTLS=YES
hostname=smtp.gmail.com:587
'root' should be replaced with your email address, and so on.  During this step, you're going to be entering in your password as raw text.  So as I mentioned in the previous section, it's safest to open a new e-mail account dedicated just for logwatching.  Save and close the file.

Test your setup
At this point everything should be up and running.  Execute the following line from your terminal (no root permissions required), and within a few seconds, you should receive an email on your designated logwatch email account.

# /usr/sbin/logwatch --mailto yourlogwatch@gmail.com

If you don't receive any e-mail, make sure there's some activity during "today" inside your /var/log/sshd/sshd.log file.  Generate some activity by (eg. by logging in and logging out remotely), and then after that, execute the above command again.

If the above command works and you do receive and e-mail, there's nothing left for you to do.  Every day at a fixed time (or whenever you power on your machine), you will receive an e-mail generated by logwatch and sent by ssmtp which will look like the example at the beginning of this post.