Thursday, September 22, 2016

intel drm driver issue and disk slowness fix in ubuntu / xbuntu 16

If you are running xubuntu of ubuntu and have a intel graphics card then the drivers are a bit broken.. This causes general slowness and the apperance of constant disk accesses and it seems to show most often after suspension wakeup.

You can confirm the issue by tailing dmesg:
dmesg -w

Your watching for this kind of message:
[20474.916101] ------------[ cut here ]------------
[20474.916133] WARNING: CPU: 3 PID: 999 at /build/linux-a2WvEb/linux-4.4.0/drivers/gpu/drm/drm_irq.c:1326 drm_wait_one_vblank+0x1b5/0x1c0 [drm]()
[20474.916135] vblank wait timed out on crtc 0
[20474.916136] Modules linked in: uas usb_storage nls_iso8859_1 mmc_block udf crc_itu_t drbg ansi_cprng ctr ccm i2400m_usb i2400m wimax intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp arc4 iwldvm uvcvideo kvm_intel kvm videobuf2_vmalloc videobuf2_memops mac80211 videobuf2_v4l2 videobuf2_core irqbypass v4l2_common videodev crct10dif_pclmul crc32_pclmul iwlwifi aesni_intel media snd_hda_codec_hdmi aes_x86_64 snd_hda_codec_realtek lrw snd_hda_codec_generic gf128mul snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq joydev glue_helper ablk_helper cryptd input_leds serio_raw snd_seq_device toshiba_acpi snd_timer sparse_keymap cfg80211 snd soundcore toshiba_bluetooth wmi toshiba_haps mei_me mei lpc_ich shpchp mac_hid tpm_infineon ip6t_REJECT
[20474.916188]  nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables parport_pc ppdev lp parport autofs4 i915 psmouse i2c_algo_bit drm_kms_helper ahci syscopyarea sysfillrect libahci sysimgblt fb_sys_fops drm sdhci_pci e1000e sdhci ptp pps_core video fjes
[20474.916224] CPU: 3 PID: 999 Comm: Xorg Not tainted 4.4.0-36-generic #55-Ubuntu
[20474.916226] Hardware name: TOSHIBA dynabook R731/36EB/Portable PC, BIOS Version 3.60   01/24/2012
[20474.916228]  0000000000000286 00000000aa071208 ffff8800a46dbb08 ffffffff813f13b3
[20474.916231]  ffff8800a46dbb50 ffffffffc00cfb38 ffff8800a46dbb40 ffffffff810810f2
[20474.916234]  ffff88014a605000 0000000000000000 0000000000000000 00000000000e0368
[20474.916237] Call Trace:
[20474.916245]  [] dump_stack+0x63/0x90
[20474.916251]  [] warn_slowpath_common+0x82/0xc0
[20474.916254]  [] warn_slowpath_fmt+0x5c/0x80
[20474.916259]  [] ? finish_wait+0x55/0x70
[20474.916272]  [] drm_wait_one_vblank+0x1b5/0x1c0 [drm]
[20474.916276]  [] ? wake_atomic_t_function+0x60/0x60
[20474.916311]  [] intel_atomic_commit+0x43a/0x6f0 [i915]
[20474.916329]  [] ? drm_atomic_set_crtc_for_connector+0x6f/0xe0 [drm]
[20474.916346]  [] drm_atomic_commit+0x37/0x60 [drm]
[20474.916359]  [] drm_atomic_helper_set_config+0x76/0xb0 [drm_kms_helper]
[20474.916374]  [] drm_mode_set_config_internal+0x62/0x100 [drm]
[20474.916389]  [] drm_mode_setcrtc+0x3cc/0x4f0 [drm]
[20474.916402]  [] drm_ioctl+0x152/0x540 [drm]
[20474.916417]  [] ? drm_mode_setplane+0x1b0/0x1b0 [drm]
[20474.916421]  [] do_vfs_ioctl+0x29f/0x490
[20474.916424]  [] ? __sb_end_write+0x21/0x30
[20474.916428]  [] ? vfs_write+0x15d/0x1a0
[20474.916430]  [] SyS_ioctl+0x79/0x90
[20474.916434]  [] entry_SYSCALL_64_fastpath+0x16/0x71
[20474.916437] ---[ end trace a5e017619a02b625 ]---

If your getting hung by it and dont what to shutdown you can get your system moving again by doing this:
sudo pm-suspend 

The full solution is the removal of the offending driver:
sudo apt-get remove xserver-xorg-video-intel

Rasberry pi samba and git server setup

Purchase and assumable physical items

* model 3 pi + case
* 32gb mirco sd card and adapter
* usb portable disk
* a usb hub (the pi lacks sufficient power output to drive a usb powered hdd)




Setup Jesse on the micro sd

Note more info at: https://www.raspberrypi.org/documentation/installation/installing-images/

1) Download the RASPBIAN JESSIE LITE .img
https://www.raspberrypi.org/downloads/raspbian/

2) Get sd card mount point

df -h | egrep "mmc|sdd"

/dev/mmcblk0p1   30G   32K   30G   1% /media/kage/3535-3133

3) umount so the card so it can be imaged

umount /dev/mmcblk0p1 

4) write img to sd card

sudo dd bs=4M if=~/Downloads/2016-05-27-raspbian-jessie-lite.img of=/dev/mmcblk0
sync

5) correct sd card partition back to max size

sudo parted /dev/mmcblk0

(parted) p free

Model: SD SL32G (sd/mmc)
Disk /dev/mmcblk0: 31.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
32.3kB  4194kB  4162kB           Free Space
1      4194kB  70.3MB  66.1MB  primary  fat16        lba
2      70.3MB  31.9GB  31.8GB  primary  ext4
31.9GB  31.9GB  15.0MB           Free Space

(parted) resizepart 2 32G                                              
(parted) p free

Model: SD SL32G (sd/mmc)
Disk /dev/mmcblk0: 31.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
32.3kB  4194kB  4162kB           Free Space
1      4194kB  70.3MB  66.1MB  primary  fat16        lba
2      70.3MB  31.9GB  31.8GB  primary  ext4

6) Eject card and reinsert. confirm mounting and sizes are ok

setup passwordless ssh for pi user (assuming the machine your on is the accessor)


1) cd to the mounted sd card and into the pi home dir

cd /media/sd-card-uuid/
pushd .
cd home/pi

2) setup .ssh with the working machines publish ssh info
mkdir .ssh
chmod 0700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
cat ~/.ssh/id_rsa.pub  >> .ssh/authorized_keys

setup static ip


1) cd to the mounted sd card and into the /etc/networks area
popd .

2) modify the interface setup
sudo vi etc/network/interface

3) setup a static hard line ip im using 192.168.1.175 feel free to adjust as needed
iface eth0 inet static
   address 192.168.1.175
   netmask 255.255.255.0
   gateway 192.168.1.1
   network 192.168.1.0
   broadcast 192.168.1.255

3b) OR setup the wifi (i would not advise this)
sudo vi etc/wpa_supplicant/wpa_supplicant.conf

# add this at the end 
network={
  ssid="network id"
  psk="network password"
}

finish up raw img adjustments


1) sync and umount card and then eject the card

sync
umount /dev/mmcblk0p1 

2) Install sd card in the pi3. You insert the micro card into the card reader on the underside of the pi3 board, face out.
Power it up and check that it seems to booted

3) Now in back your work machine find out where it went

nmap -sP 192.168.1.0/24

4) If it booted at the correct address then great otherwise check your router setups

check access and secure the default password


1) ssh into the pi. Note the default password is "raspberry" but you shouldnt need it your ssh key is the access method.

ssh pi@192.168.1.175

2) Scramble the default password (I use a software vault, keepassx, to generate and store passwords) this way if someone gets in they cant just sudo with the default password and get root as well

passwd

clean up and secure the ssh access


1) Its an image so /etc/ssh identity files are already setup.. thats not very healthy lets rebuild them

sudo rm -f /etc/ssh/*key*
sudo dpkg-reconfigure openssh-server

2) Confirm the ssh is ok (for chmods etc)

ls -al ~/.ssh/authorized_keys

3) Now secure the ssh access to keys only

sudo vi  /etc/ssh/sshd_config

4) Edit/add the following lines

PasswordAuthentication no
 AllowTcpForwarding no
 X11Forwarding no

5) And confirm that ssh still works (with a second window).. Note if you make a mistake you can always turn off the pi, eject the sd card and edit it directly in your working PC.

update machine and fix various other image guff


1) first update the images software.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

2) after that I seemed to be getting this mess with the locale being bad..

perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),

3) so fix it with:

sudo dpkg-reconfigure locales

Setup usb disk

for more info refer to: http://www.howtogeek.com/139433/how-to-turn-a-raspberry-pi-into-a-low-power-network-storage-device/

1) find out what usb disk we have attached

sudo blkid 

/dev/sda1: LABEL="EC-PHU3" UUID="2E24054E24051B0B" TYPE="ntfs" PARTUUID="8418c874-01"

2) Right looks like a ntfs (but it could have also been vfat etc). So setup ntfs drivers

sudo apt-get install ntfs-3g 

3) make the mount point

sudo mkdir       /media/hdd
sudo chmod a+rwx /media/hdd

4) And check that it can actually mount

sudo mount -t auto /dev/sda1 /media/hdd

4a) ok.. stupid things are happening

modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.11-v7+/modules.dep.bin'
ntfs-3g-mount: fuse device is missing, try 'modprobe fuse' as root

4b) well thats because.. it has the wrong /lib/modules version installed!!.. wtf!

ls /lib/modules/4.4.13-v7+/

4c) So after googling for a while turns out there is a surprising fix... most likely it was something in that first apt-get update cycle caused all the modules to move forward!

sudo reboot

pi@raspberrypi:/lib/modules $ uname -a
Linux raspberrypi 4.4.13-v7+ #894 SMP Mon Jun 13 13:13:27 BST 2016 armv7l GNU/Linux

4d) But something crazy happened to the ntfs-3g had to install it again??? no idea why..

sudo apt-get install ntfs-3g

5) now mount and test access

sudo mount -t ntfs-3g -o uid=pi,gid=pi /dev/sda1 /media/hdd
cd /media/hdd
touch abc.txt
ls 
rm /media/hdd/abc.txt

6) if that worked and u can see the test file move on otherwise start googling..

Setup usb disk at permanent location


1) now we are going to make the usb disk a permanent fixture, umount it

sudo umount /media/hdd

2) update /etc/fstab. We are going make all files owned by user pi and group users, read and writable
extra info at
* https://www.howtoforge.com/reducing-disk-io-by-mounting-partitions-with-noatime
* http://raspi.tv/2012/how-to-mount-and-use-a-usb-hard-disk-with-the-raspberry-pi
* http://www.omaroid.com/fstab-permission-masks-explained/
* note nobootwait doesnt work???

sudo vi etc/fstab

2a) and add the following line
/dev/sda1       /media/hdd    ntfs-3g noatime,umask=0,uid=pi,gid=users,dmask=0007,fmask=0117 0 0

2b) or you can use something like:
/dev/sda1 /media/hdd vfat uid=pi,gid=users,umask=0022,sync,auto,nosuid,rw,nouser 0 0

3) and then mount it
sudo mount /dev/sda1

4) test access

touch /media/hdd/abc.txt
rm /media/hdd/abc.txt

setup samba for remote share


1) ok build samba share location...

mkdir /media/hdd/share

2) install samba

sudo apt-get install samba samba-common-bin libpam-smbpass

3) Configure samba

sudo cp /etc/samba/smb.conf /etc/samba/smb.conf.20160922
sudo vi /etc/samba/smb.conf

4) insert or uncomment the following in the Authorization section

security = user
unix password sync = yes

5) comment out homes,printers sections.. its junk we dont want or need

6) insert "share" section

[Share]
  comment = 1TB_samba
  path = /media/hdd/share
  valid users = remote
  read only = No

7) bounce the samba service

sudo /etc/init.d/samba restart

8) check its setup

testparm -s

9) now in the working pc with what ever file manager u use "browse" your network and locate the pi likely called "raspberrypi" open it and confirm there are no "printers" stuff and "share" is visible

10) try to access the "share" using "guest" access, then as the user "pi" with the default password, your new password and confirm all of this has no access

11) back in the pi. Make the "remote" user and passwd it. Again password vault scramble a password for it.. but make certain its type-able
For more info refer to http://raspi.tv/2012/how-to-create-a-new-user-on-raspberry-pi

sudo adduser remote
sudo usermod -a -G users remote
groups remote
sudo smbpasswd -e remote

12) then bounce samba

sudo /etc/init.d/samba restart

13) then back in the working PC retest the samba share using the user "remote" and there unix password and confirm file/dir create/delete etc. Also check the guest and "pi" users again to be certain..

14) then reboot and check mount stays put and samba works over the reboot

sudo reboot

Setup the git ssh server


1) make the repos storage location

mkdir /media/hdd/repos

2) install git

sudo apt-get install git-core

3) add a git user

sudo adduser git
sudo usermod -a -G users git
groups git

4) switch over to the new "git" user

#cat *pi* users access keys first so u can copy it later for step 6
cat .ssh/authorized_keys

su git
cd ~

5) confirm sudo limits.. (should fail)

sudo ls

6) setup the "git" users ssh access

mkdir .ssh
chmod 0700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
vi ~/.ssh/authorized_keys

7) from your working pc confirm ssh access

ssh git@192.168.1.175

8) link the repos storage to the git users home

ln -s /media/hdd/repos repos

9) create a test repo

cd ~/repos 
git init --bare test.git

9a) Note ignore chmod errors (we have it forced to a certian way in fstab)
error: chmod on /media/hdd/repos/test.git/config.lock failed: Operation not permitted
error: chmod on /media/hdd/repos/test.git/config.lock failed: Operation not permitted

Note to self. There might be an issue with executable scripts loosing x permission due to this...may have to rethink this.

10) Then on your working machine clone and check that the repo worked

git clone git@:~/repos/test.git 
cd test/
ls
touch test.txt
git add test.txt 
git commit -a -m "Initial"
git push

11) then reclone and confirm test.txt is there in the new clone

cd ..
git clone git@192.168.1.4:~/repos/test.git test2
cd test2
ls 

setup automatic updating


now we know all the basics are working... lets harden it a bit

1) setup auto updates
for more info see https://unixsysdoc.wordpress.com/2015/05/18/xubuntu-security-and-hardening-part-i-basics/

sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

disable needless hardware


1) disable wifi and bluetooth (as im using the hard line)

sudo touch /etc/modprobe.d/disable_rpi3_wifi_bt.conf
sudo vi /etc/modprobe.d/disable_rpi3_wifi_bt.conf

2) add the lines

##wifi
blacklist brcmfmac
blacklist brcmutil
##blue tooth
blacklist btbcm
blacklist hci_uart

3) and reboot

sudo reboot

4) log back in to the pi and confirm wifi is off

ifconfig

setup the firewall

for more info check; https://www.digitalocean.com/community/tutorials/how-to-setup-a-firewall-with-ufw-on-an-ubuntu-and-debian-cloud-server

1) install ufw.. its an easy to use firewall

sudo apt-get install ufw 

2) let in local ssh

sudo ufw allow ssh

3) let in local samba (and bonjour)
for more info check: https://ubuntuforums.org/showthread.php?t=806000

sudo ufw allow proto tcp to any port 135 from 192.168.0.0/16
sudo ufw allow proto udp to any port 137 from 192.168.0.0/16
sudo ufw allow proto udp to any port 138 from 192.168.0.0/16
sudo ufw allow proto tcp to any port 139 from 192.168.0.0/16
sudo ufw allow proto tcp to any port 445 from 192.168.0.0/16
sudo ufw allow proto udp to any port 5353 from 192.168.0.0/16 

4) and turn it on

sudo ufw enable

5) then retest all the samba acces and git cloning

Setup the hostname to something better


1) adjust the hostname so you know which machine your in replace "raspberrypi" with the name you want

sudo vi /etc/hostname
sudo vi /etc/hosts

2) and reboot

sudo reboot

Clean up


1) clean up the working pc git clones

rm -rf test
rm -rf test2

2) go back to the pi clean out the test and import/setup your real ones

rm test.git

bonus notes..


1) i trialed wake on lan

sudo apt-get install ethtool
ethtool eth0 | grep wake
ethtool -s eth0 wol g

# it responds with bad news
Cannot get wake-on-lan settings: Operation not permitted