pimail ICT4D bridging the digital divide

pimail · by **paulv** » Sat May 18, 2013 1:26 pm

I use one of my Pi's (Pies?) in a more or less Mission Critical Application on another continent, and was getting concerned with the reports about SD cards getting corrupted. Apparently, these cards are not really made for heavy read/write applications, although nice and cheap in educational and hobby environments.

If you want to offload the main filesystem to another medium, there are several examples if you Google around enough. Unfortunately, some are not really applicable to the Raspberry Pi, others are for super-super users, who know what they are doing.

I created this description to make it also usable for power-users, although I am by no means a super-power-user myself. Beware! :roll:

To make it simple and easier, I created two parts to the procedure.
1. For the simple use of the Pi with only 1 extra USB drive or stick
2. An extended version if you need more than one drive attached to your Pi. (Raspbmc users?) Note that this version is a little more complicated.

We will start off with a brand new SD card, and a brand new USB stick.
The size and speed of the SD card is no longer important and any card over 2G will do. (it has to be 2G because we're going to load complete Debian on it)
The size of the USB drive/stick depends on your application, anything over 2G will do. Speed matters a bit more here, but I'm not going for a speed improvement, that's another topic.

In this description I will use a micro SDHC memory card with 4GM, class 10, because I use the cards for other applications as well and I don't like the SD card sticking outside of the Pi cover (already broken one SD card adapter). And I also use a micro USB stick with a size of 8GB.

First of all, you need to load the Wheezy Raspbian image (I used 2013-02-09) on the SD card. There are enough procedures to tell you how to do that. I use win32diskimager on my W7 laptop.
Once that is done, you can put it in your Pi and boot it up.
If you use a LAN cable, you're good to go. If you run a headless Pi (like I do) over WIFI, you need to configure that first, again, there are many procedures on how to do that for your particular dongle. I use an EDIMAX in this particular setup. Obviously, I'm using ssh, but that does not matter.

First of all, you should run raspi-config and setup ONLY the following:

Code: Select all: sudo raspi-config

configure_keyboard (not needed for a headless Pi)
change_pass (change the raspberry password if you wish at this moment, can also be done later)
change_locale
change_timezone
memory_split (a headless can be set to 16MB)
Do not update/upgrade at this moment! That will cause a lot of thrashing on the SD card, which is what we want to avoid.

After a reboot to load the new parameters, we're good to go.

Check the filesystem:

Code: Select all: pi@raspberrypi ~ $ df -h Filesystem Size Used Avail Use% Mounted on rootfs 1.8G 1.4G 247M 86% / /dev/root 1.8G 1.4G 247M 86% / devtmpfs 235M 0 235M 0% /dev tmpfs 49M 248K 49M 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 98M 0 98M 0% /run/shm /dev/mmcblk0p1 56M 19M 38M 34% /boot pi@raspberrypi ~ $

Notice that although I use an 8GB SD card, the default is 1.8G in size, because we did not use expand_rootfs.
The entry /dev/mmcblk0p1 is the boot partition (p1) on device mmcblk0, the SD card.

The boot partition will be the only piece on the SD card used when we're done, and that is because of the particular way the Pi goes through the boot process. It actually starts with the graphics chip loading this data first, and it then releases the reset pin of the CPU which continues and takes over.

Now plug in your USB drive/stick if you have not done that yet (I'll refer to stick from now on)

Check to see if it's recognized:

Code: Select all: pi@raspberrypi ~ $ lsusb Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. Bus 001 Device 004: ID 7392:7811 Edimax Technology Co., Ltd EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS] Bus 001 Device 006: ID 8644:800b pi@raspberrypi ~ $

Device 006 is the USB adapter. The ID shows a unique manufacturer identifier.
Also check to see if the OS recognized the device: (you can also use the command dmesg)

Code: Select all: pi@raspberrypi ~ $ tail /var/log/messages May 18 11:44:31 raspberrypi kernel: [ 64.093866] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3 May 18 11:44:31 raspberrypi kernel: [ 64.093883] usb 1-1.3: Product: Intenso Micro Line May 18 11:44:31 raspberrypi kernel: [ 64.093898] usb 1-1.3: Manufacturer: Intenso May 18 11:44:31 raspberrypi kernel: [ 64.093912] usb 1-1.3: SerialNumber: 12390000000004E2 May 18 11:44:31 raspberrypi kernel: [ 64.112228] scsi1 : usb-storage 1-1.3:1.0 May 18 11:44:32 raspberrypi kernel: [ 65.113335] scsi 1:0:0:0: Direct-Access Intenso Micro Line 1.01 PQ: 0 ANSI: 2 May 18 11:44:32 raspberrypi kernel: [ 65.114862] sd 1:0:0:0: [sda] 15669248 512-byte logical blocks: (8.02 GB/7.47 GiB) May 18 11:44:32 raspberrypi kernel: [ 65.115607] sd 1:0:0:0: [sda] Write Protect is off May 18 11:44:32 raspberrypi kernel: [ 65.126062] sda: sda1 May 18 11:44:32 raspberrypi kernel: [ 65.129907] sd 1:0:0:0: [sda] Attached SCSI removable disk pi@raspberrypi ~ $

Yep, the USB stick can be referenced as /dev/sda

There is no way of knowing what name the OS assigns to additional USB drives, if that's what you want to do, you need to follow the extended version of the procedure down below.

If one USB memory stick is what you're going to use, start here:

Create a filesystem on the USB stick, use "p" to list the partitions

Code: Select all: pi@raspberrypi ~ $ sudo fdisk /dev/sda Command (m for help): p Disk /dev/sda: 8022 MB, 8022654976 bytes 94 heads, 14 sectors/track, 11906 cylinders, total 15669248 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 32 15669247 7834608 b W95 FAT32 Command (m for help):

There is a partition, formatted for Windows. We need to reformat the drive for Linux.
Enter d to delete partion(s).
Note
If you are going to use the stick only for the Debian filesystem, go on. If you have a large stick or a USB drive, you may want to consider making one partition for Debian, and other ones for whatever you like. This is beyond the scope of this article, but I'm sure you can figure it out.
We will continue with the simple solution, one partition.

Enter n to create a new primary partition ( number 1) and use the full capacity by hitting return a few times until done:

Code: Select all: Command (m for help): d Selected partition 1 Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-15669247, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-15669247, default 15669247): Using default value 15669247 Command (m for help):

Now write the new information to the stick by entering "w":

Code: Select all: Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. pi@raspberrypi ~ $

Done, so now we can format the partition:

Code: Select all: pi@raspberrypi ~ $ sudo mke2fs -t ext4 -L rootfs /dev/sda1 mke2fs 1.42.5 (29-Jul-2012) Filesystem label=rootfs OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 490560 inodes, 1958652 blocks 97932 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=2009071616 60 block groups 32768 blocks per group, 32768 fragments per group 8176 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done pi@raspberrypi ~ $

And we need to mount the partition to start using it:

Code: Select all: pi@raspberrypi ~ $ sudo mount /dev/sda1 /mnt pi@raspberrypi ~ $

Let's see what we've done so far:

Code: Select all: pi@raspberrypi ~ $ df -h Filesystem Size Used Avail Use% Mounted on rootfs 1.8G 1.4G 247M 86% / /dev/root 1.8G 1.4G 247M 86% / devtmpfs 235M 0 235M 0% /dev tmpfs 49M 248K 49M 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 98M 0 98M 0% /run/shm /dev/mmcblk0p1 56M 19M 38M 34% /boot /dev/sda1 7.4G 145M 6.9G 3% /mnt pi@raspberrypi ~ $

We made the stick /dev/sda1 available as /mnt for Linux, so now we can start to use it.

We can copy (mirror) the data in /dev/root to the new drive.
We can use dd, but that is a bit copy program and because it also copies the empty space on the SD card, it is very, very slow.
Better is to use rsync.
Install that first:

Code: Select all: pi@raspberrypi ~ $ sudo apt-get install rsync ... pi@raspberrypi ~ $

And run it. Although much faster, it will still take some time to create a mirror image of the file system on the USB drive. We will copy everything from "/" to the just mounted drive /mnt (which is /dev/sda1)

Code: Select all: pi@raspberrypi ~ $ sudo rsync -axv / /mnt ...

On my system it took about 15 minutes. Make sure this process does not get interrupted!
If you want, you can open another ssh session and continue with the process, so you don't have to wait for the rsync process to complete.

We're going to tell the Pi where to find the new data when it boots Debian.
The file we need to change is /boot/cmdline.txt
Before we're going to do that, we're going to create a backup copy so we can change things back if there is a problem. Be careful, if you mess up this file, you will almost certainly stop the Pi from booting, and you may have to start all over again by reloading the image.

Code: Select all: sudo cp /boot/cmdline.txt /boot/cmdline.orig

The contends of this file is one very long command string, and we need to replace the reference to the partition on the SD card (root=/dev/mmcblk0p2) to the new USB stick (root=/dev/sda1)
Here is a copy of the original file:

Code: Select all: pi@raspberrypi ~ $ cat /boot/cmdline.txt dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait pi@raspberrypi ~ $

To be on the safe side, there is one more command we should add. During the power-up/boot sequence, it may take the USB stick a little longer to be ready, than the OS expects. If we add a delay of 5 seconds (rootdelay=5), the USB stick will be ready when it's needed. You may not need it, or a smaller amount, so go figure it out. When my remote system has to reboot, I want it to be safe, a few seconds more is not an issue.

Use your favorite editor to add these two changes like so:

Code: Select all: sudo nano /boot/cmdline.txt

It should look like this:

Code: Select all: pi@raspberrypi ~ $ cat /boot/cmdline.txt dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait rootdelay=5 pi@raspberrypi ~ $

Next we need to enter a few things into the filesystem table, /etc/fstab. (on the stick, not the SD card)

Code: Select all: sudo nano /mnt/etc/fstab

Add the following line :

Code: Select all: /dev/sda1 / ext4 defaults,noatime 0 1

And put a comment in front of the reference to the SD card:

Code: Select all: #/dev/mmcblk0p2 / ext4 defaults,noatime 0 1

It should now look like this:

Code: Select all: proc /proc proc defaults 0 0 /dev/sda1 / ext4 defaults,noatime 0 1 /dev/mmcblk0p1 /boot vfat defaults 0 2 #/dev/mmcblk0p2 / ext4 defaults,noatime 0 1 # a swapfile is not a swap partition, so no using swapon|off from here on, use dphys-swapfile swap[on|off] for that

This is it for the simple version. you can now reboot the system:

Code: Select all: sudo reboot

Note that you will not see much activity on the ACT LED anymore. This LED only shows R/W activity on the SD card.
After the Pi has booted, check the filesystem:

Code: Select all: pi@raspberrypi ~ $ df -h Filesystem Size Used Avail Use% Mounted on rootfs 7.4G 1.6G 5.5G 22% / /dev/root 7.4G 1.6G 5.5G 22% / devtmpfs 235M 0 235M 0% /dev tmpfs 49M 248K 49M 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 98M 0 98M 0% /run/shm /dev/mmcblk0p1 56M 19M 38M 34% /boot pi@raspberrypi ~ $

The rootfs is now on the stick, notice the size of 7.5G

Changing the swap file size
There is no need to mess with swap partitions on the Pi, this is implemented differently from other Linux systems. Raspbian uses dphys-swapfile, which is a swap-file based solution instead of the "standard" swap-partition solution. It is also much easier to change the size of the swap.

If you need to change the swap file size, proceed as follows:

Look at the swap data first by using the top command:

Code: Select all: pi@raspberrypi ~ $ top top - 13:28:20 up 3 min, 1 user, load average: 0.18, 0.37, 0.18 Tasks: 66 total, 1 running, 65 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem: 497544 total, 55700 used, 441844 free, 5432 buffers KiB Swap: 102396 total, 0 used, 102396 free, 27592 cached

102396 is 100MB of swap space

The configuration file for the swap space is: /etc/dphys-swapfile

The content is very simple. By default Raspbian has 100MB of swap space:

Code: Select all: pi@raspberrypi ~ $ cat /etc/dphys-swapfile CONF_SWAPSIZE=100 pi@raspberrypi ~ $

If you want to change the size, you need to modify the number and then resize and restart the dphys-swapfile:

Code: Select all: pi@raspberrypi ~ $ sudo nano /etc/dphys-swapfile pi@raspberrypi ~ $

I changed the size from 100 MB to 512MB, the full amount of RAM.
After the parameter in the file has been changed run:

Code: Select all: pi@raspberrypi ~ $ sudo dphys-swapfile setup want /var/swap=512MByte, checking existing: deleting wrong size file (104857600), generating swapfile ... of 512MBytes pi@raspberrypi ~ $

To load the new size, we need to stop the swapfile process:

Code: Select all: pi@raspberrypi ~ $ sudo /etc/init.d/dphys-swapfile stop Stopping dphys-swapfile swapfile setup ..., done. pi@raspberrypi ~ $

And restart it:

Code: Select all: pi@raspberrypi ~ $ sudo /etc/init.d/dphys-swapfile start Starting dphys-swapfile swapfile setup ... want /var/swap=512MByte, checking existing: keeping it done. pi@raspberrypi ~ $

And now we can check it again with top:

Code: Select all: top - 13:37:22 up 12 min, 1 user, load average: 0.01, 0.08, 0.12 Tasks: 64 total, 1 running, 63 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.7 us, 0.3 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 497544 total, 57016 used, 440528 free, 5556 buffers KiB Swap: 524284 total, 0 used, 524284 free, 28604 cached

Done!
Now that you have everything running on the USB stick, it's time to update and upgrade Debian:

Code: Select all: sudo apt-get update sudo apt-get upgrade -y sudo apt-get autoremove

And off you go...

Changing the process back to the SD card
If you ever run into problems booting from the USB stick, you can try to move the original /boot/cmdline.txt back by putting the SD card back into your PC, and use the Filemanager to copy the file /boot/cmdline.txt to something else (/boot/cmdline.usb) and copy the name of /boot/cmdline.orig to /boot/cmdline.txt. (watch out: do not use rename, because Windows may rename the file called cmdline.txt to cmdline.txt.txt !)
If your system still works with the USB stick, you can also mount the windows partition on the SD card and make the changes to the cmdline files from within Debian.

Before you do that however, remember that we modified the /etc/fstab file on the SD card and then moved it to the USB stick. We need to change the contends back on the SD card to the original settings before we can easily change the filesystem back to the SD card.
If you followed the procedure above, do the following:

Code: Select all: pi@raspberrypi ~ $ df -h Filesystem Size Used Avail Use% Mounted on rootfs 7.4G 2.5G 4.6G 36% / /dev/root 7.4G 2.5G 4.6G 36% / devtmpfs 235M 0 235M 0% /dev tmpfs 49M 240K 49M 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 98M 0 98M 0% /run/shm /dev/mmcblk0p1 56M 19M 38M 33% /boot

Use the device name without the partition (p1)

Code: Select all: pi@raspberrypi ~ $ sudo fdisk /dev/mmcblk0 Command (m for help): p Disk /dev/mmcblk0: 4035 MB, 4035969024 bytes 4 heads, 16 sectors/track, 123168 cylinders, total 7882752 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00014d34 Device Boot Start End Blocks Id System /dev/mmcblk0p1 8192 122879 57344 c W95 FAT32 (LBA) /dev/mmcblk0p2 122880 7882751 3879936 83 Linux Command (m for help): q

So now that we know what the Linux file system partition is (p2), we can mount it.

Code: Select all: pi@raspberrypi ~ $ sudo mount /dev/mmcblk0p2 /mnt

And check it

Code: Select all: pi@raspberrypi ~ $ df -h Filesystem Size Used Avail Use% Mounted on rootfs 7.4G 2.5G 4.6G 36% / /dev/root 7.4G 2.5G 4.6G 36% / devtmpfs 235M 0 235M 0% /dev tmpfs 49M 240K 49M 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 98M 0 98M 0% /run/shm /dev/mmcblk0p1 56M 19M 38M 33% /boot /dev/mmcblk0p2 3.7G 1.7G 1.9G 47% /mnt

OK, now we can edit the fstab file on the SD card so it looks like the original version again.

Code: Select all: pi@raspberrypi ~ $ sudo nano /mnt/etc/fstab

And check it.

Code: Select all: pi@raspberrypi ~ $ sudo cat /mnt/etc/fstab proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults 0 2 /dev/mmcblk0p2 / ext4 defaults,noatime 0 1 # a swapfile is not a swap partition, so no using swapon|off from here on, use dphys-swapfile swap[on|off] for that #filesystem on USB stick pi@raspberrypi ~ $

With a bit of luck, by just doing a copy of the right /boot/cmdline version on the Windows partition of the SD card, you will be able to change the booting process from the SD card. If not, there is enough information in this procedure to help you out I hope.

Extended procedure:

Because we must force Debian to boot from the right USB device, we need to assign unique identifiers for the partition and the filesystem, rather than the names the system gives to the USB devices, like /dev/sda.

I'm assuming that after reading the above, we can be a little more terse.
Instead of using the standard fdisk command, we will use the more modern version gdisk.
We can install gdisk with:

Code: Select all: sudo apt-get install gdisk

We start the partitioning with:

Code: Select all: sudo gdisk /dev/sda

And we actually use the same sequence as with fdisk. First "d" for delete, then "n" for new and "w" to write it all out. Again use the defaults by hitting return. If you want more partitions, now is the time.

When you use the "i" command (information), you can see the following details:

Code: Select all: pi@raspberrypi ~ $ sudo gdisk /dev/sda GPT fdisk (gdisk) version 0.8.5 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): i Using 1 Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem) Partition unique GUID: 2C7FD5E5-5CC4-434C-BFF6-F45336F7140B First sector: 2048 (at 1024.0 KiB) Last sector: 15669214 (at 7.5 GiB) Partition size: 15667167 sectors (7.5 GiB) Attribute flags: 0000000000000000 Partition name: 'Linux filesystem' Command (? for help):

And this is what we need: Partition unique GUID: 2C7FD5E5-5CC4-434C-BFF6-F45336F7140B
This information (PARTUUID) goes into the /boot/cmdline.txt like so:

Code: Select all: pi@raspberrypi ~ $ cat /boot/cmdline.txt dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=PARTUUID=2C7FD5E5-5CC4-434C-BFF6-F45336F7140B rootfstype=ext4 elevator=deadline rootwait rootdelay=5 pi@raspberrypi ~ $

Don't forget to add the rootdelay!

We now continue with formatting, mounting, installing rsync and the mirroring:

Code: Select all: sudo mke2fs -t ext4 -L rootfs /dev/sda1 sudo mount /dev/sda1 /mnt sudo apt-get install rsync sudo rsync -axv / /mnt

Now we need to get a unique identifier for the fstab drive information:

Code: Select all: pi@raspberrypi ~ $ sudo tune2fs -l /dev/sda1 tune2fs 1.42.5 (29-Jul-2012) Filesystem volume name: rootfs Last mounted on: / Filesystem UUID: 3d2cb3c9-7e2b-4255-ac63-863bd3ad7993 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) ... pi@raspberrypi ~ $

And this is what we need: Filesystem UUID: 3d2cb3c9-7e2b-4255-ac63-863bd3ad7993

Next we can enter the information into the filesystem table, /etc/fstab (on the stick, not the SD card). Add the following line:

Code: Select all: /dev/disk/by-uuid/3d2cb3c9-7e2b-4255-ac63-863bd3ad7993 / ext4 defaults,noatime 0 1

And put a comment in front of the reference on the SD card:

Code: Select all: #/dev/mmcblk0p2 / ext4 defaults,noatime 0 1

And that's it. If you are done checking everything (don't mix up the partition GUID and the filesystem UUID!!!) , you can reboot now and check if all is well.

Have fun!

[edit] I have made the changes based on the input from violagirl23 on sept-26-2013. Thank you girl!

pimail ICT4D bridging the digital divide

Dienstag, 31. März 2015

HOWTO: Moving Raspberry Pi 2 filesystem to a USB.stick or USB.harddisk

Keine Kommentare:

Kommentar veröffentlichen

Gesamtzahl der Seitenaufrufe