Friday, September 5, 2014
Saturday, August 30, 2014
Image Credit:" Raspberry Pi B+ top" by Lucasbosch - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons
Hi friends, I have made changes to the original article and scripts on github to support NOOBs SD cards. These SD cards can contain one or more raspberry PI operating systems. My scripts will backup one of those (linux based) OSs and allow you to restore it to a fresh SD. Note, it will only backup whichever OS is booted, and only that OS. During the restore script run, it will grow the partitions to fill the entire SD card. I have tested it with Raspbian and Pidora. Thanks and Enjoy!
Quick note of thanks to @KevinSidwar who pointed this out to me, I had never used NOOBs and it's an important feature for my scripts to support since a lot of new Rapsberry Pi come with a NOOBs card these days. Thanks Kevin!
You can find the article at: http://sysmatt.blogspot.com/2014/08/backup-restore-customize-and-clone-your.html
Sunday, August 17, 2014
Ok, so it will not solve the next quantum physics conundrum, but it might just ignite a spark in the next comp sci or physics genius. This is "Tiny Titan" designed at Oak Ridge National Laboratory specifically to get students thinking parallel. Writing software for massive high performance computing (HPC) systems is no easy task. With the simple hardware design built from low cost $35 Raspberry Pi boards and example projects a student can get their feet wet. On the Tiny Titan github there is example code in C and Python and lots of documentation.
Tiny Titan: https://www.olcf.ornl.gov/2014/06/02/titans-tiny-counterpart-engages-educates/
PrefaceIt's been a long time since I originally wrote this article back in 2005 for Linux Journal. Do people even read paper any more? *smirk* I figured it needed a little update. I will walk you through all the steps necessary to boot up ancient UNIX (ancient as in 1970s) on your very own computer using the SIMH simulator. I also include special notes if you want to run the examples on the Raspberry Pi and Raspbian. (Which didn't even exist in 2005!). Enjoy!
I have always had a voracious appetite for early minicomputer and mainframe history. I believe recorded history itself is the single-most important innovation of human existence. We humans seem to have a hard-wired compulsion to record, pass on and learn from the mistakes and successes of those before us. Open-source software and hardware is the natural evolution of this concept applied to technology. In the Open Source philosophy, we are all free to learn from the wealth of software created by the masses that came before us. By examining the evolution of a project, we can learn from the mistakes of others and, perhaps most important, copy verbatim from their successes. By harnessing this freely available history as well as unfettered cooperation, we advance the common good.
Recently, some companies have begun to loosen their grip on the early computing “intellectual property”. Although some have not fully embraced open source, these sometimes small, token gestures offer us a wealth of knowledge. In this article, I focus on how we can explore early operating system history by running “historic” UNIX releases on our very own Linux boxes using a simulator. The SCO Group (Yes, “them”, previously Caldera, Inc.) claims current ownership of early UNIXes and has released them under an “Ancient Unix” license, which allows for noncommercial use.
I focus here on the UNIX V5 release, because it is the earliest available UNIX. UNIX V6, V7 and various early BSD releases are also available. If you plan on trying out any of these OSes, examine the licenses included with each before booting them up.
Stranger in a Strange Land: the UNIX V5 User Environment
The UNIX V5 system provided in the disk image we will use is rather stark and unfriendly compared to modern, lush UNIX/Linux systems. Here are a few pointers before we get started:
- sh is the shell. It's only 858 lines of C; don't expect it to work like the /bin/bash shell.
- Use chdir to change the default directory.
- Backspace and arrow keys rarely work.
- ed is the text editor; see en.wikipedia.org/wiki/Ed.
- bas is a basic interpreter.
- fc is a FORTRAN interpreter.
- cc is the C compiler.
- Source code is in /usr/source.
- There are not many files (528), so use find / -print to see what else is included. It should scroll by just fast enough to read it ;-)
SIMH is a ground-up system simulator; it simulates the CPU, memory, firmware and devices of a number of early computer systems. This means that original distributed software can run unmodified on these simulated systems. SIMH successfully simulates devices such as disks, tape drives, printers and networking devices. This means that not only can we run these historic systems, but we can communicate and transfer data to and from them using modern technologies and protocols. A great deal of thanks is owed to the authors and contributors of SIMH. Their decision to release under open source furthers all our understanding of our history and guarantees that this history will always be free.
Getting Started: Installing SIMH
Raspberry Pi Note|
If you are running this on a Raspberry Pi running Raspbian you are in luck, SIMH is available as pre-built software. Skip the compile in this section and just install simh using apt-get:
sudo apt-get install simh
Then just skip to Running UNIX V5
Below I will detail the command line steps needed to download and compile the SIMH source code. The latest SIMH release is V3.9-0 at the time of this writing.
If you want to use Ethernet emulation, you will need to install the PCAP development packages for your OS. The SIMH installation documents explain how to do this, and you can skip this step if you're not going to be using networking support on your simulated machines. If you are running Fedora or similar you can simply run:
The stuff you type to compile SIMH is below in bold.
This builds all possible system simulators. Each simulator becomes a separate binary in the ./BIN/ directory. SIMH can be run as any normal user, but if you want to use Ethernet network simulation, you need to execute it as root (under UNIX) to allow libpcap access to the Ethernet device. Otherwise, continue as any non-root user. Our particular example of UNIX V5 does not use networking.
Running UNIX V5
Raspberry Pi Note|
If you are running this on a Raspberry Pi running Raspbian you can perform all these steps as the "pi" user in it's home directory. Make sure you have at least 4MB of free space. (Yes, You read the correct. The entire UNIX V5 is less then 4MB.)
UNIX V5, released in June 1974, was still very early in UNIX development at Bell Labs. Much of the system was still written in assembler. This disk image includes a working C compiler (cc) and a great deal of interesting source code under /usr/source.
To begin our exploration, we must download the UNIX V5 disk image. In the same shell as you compiled SIMH, run the following:
This zip archive contains the pre-installed image file as well as a README and file containing license information. The disk image is a snapshot of a working installed system. In this case, it is simulating an RK05 disk drive. We must now collect the pieces we need to get this system booted. Begin by copying the BIN/pdp11 binary from under the SIMH build into our uv5 directory, then we will create a small config file "pdp11.ini" that tells the simulator how to boot UNIX.
Raspberry Pi Note|
If you are running this on a Raspberry Pi running Raspbian skip the next step where you copy (cp) the pdp11 binary and instead, run the following:
ln -s /usr/bin/pdp11 .
Create a file pdp11.ini with your favorite editor and place the following lines in it.
This tells the simulator what kind of CPU to emulate and to attach the unix_v5_rk.dsk file as a simulated RK-style disk using the rk0 device name. Finally, this file tells the simulator to boot the OS image on that disk.
Your uv5 simulator directory should look like the following:
We are ready. Let's boot UNIX V5 and have some fun! Follow the example session below or go off on your own exploring. Note, you will likely need to use CTRL-h as backspace if you make a mistake.
Let's look at some source code, starting with the "echo" program. It's small and simple.
Let's compile the echo program, rename it and run a "Hello World"
Nice! Now, let's create, compile and run our own hello world C program. We will use "cat" as a poor man's editor to create the source file.
Let's try a basic program next, use cat to create hello.b with some basic source
All done for now, Let's HALT the system. In true historical UNIX style, we will run sync three times, then push the HALT button. Or in our case, the SIMH equivalent which is CTRL-e
That's it! You're up and running. You have officially set your fingers on a “real” historical UNIX system. As you can see, there is plenty of source code to look over and a working compiler to play with. UNIX V5 is only one of the early operating systems you can explore with SIMH. On the SIMH Web site, you will find a repository of disk images for lots of other systems. The learning curve can be quite steep on these older OSs but there is a lot to learn from them. You can find conceptual bit and pieces of them strewn about inside all modern systems. It's living history at your fingertips.
If you are interested in seeing what a PDP-11 system and RK05 disk actually looked like, take a look at the photo gallery on the SIMH Web site (see Resources). Also, try searching Google Images for a wealth of great photographs.
- SIMH Web Site: simh.trailing-edge.com
- SIMH Users Guide: http://simh.trailing-edge.com/pdf/simh_doc.pdf
- SIMH Photo Gallery: simh.trailing-edge.com/photos.html
- SIMH Contributors: simh.trailing-edge.com/contrib.html
- PDP-11 History and Photos: www.pdp11.org
- UNIX History Timeline (Printable!): www.levenez.com/unix/
- Open Group UNIX History: www.unix.org/what_is_unix
- The Unix Heritage Society: www.tuhs.org
- PDP Unix Preservation Society: minnie.tuhs.org/PUPS
About the authorMatthew is the Senior Enterprise Architect at the New Jersey Institute of Technology (NJIT). When he is not working, he enjoys hacking on electronics, microcontrollers and embedded systems. He can also make a mean falafel when the circumstances call for it. Matthew is a total nerd and thus completely useless in conversations regarding "sports". Matthew also greatly dislikes talking about himself in the third person in author biographies.
For more random nonsense from Matthew he suggests following him on twitter: @sysmatt
Like this article? See more of Matt's published works
Wednesday, August 13, 2014
The response has been really cool. It got picked up by Hackaday and Lifehacker. It was also re-tweeted by Adafruit and a dozen others. And heck of a lot more feedback when when my stuff ended up on dead trees.
Anyway, this was sort of a ramble, but i think there will be more posts like the last one. It's been a ride!
Saturday, August 9, 2014
Wednesday, August 6, 2014
Joshua Wise accidentally destroys his SD card, rather be content with data loss and move on -- He pulls out a FPGA devboard and de-solders the NAND chip. This great blog post walks us through the entire process from smashed SD card to files recovered.
Backup, Restore, Customize and Clone your Raspberry Pi SD Cards (tutorial) [Updated 8/30 for NOOBs SD]
Image Credit:" Raspberry Pi B+ top" by Lucasbosch - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons
NOOBs Users: Note, My scripts on github have now been updated to support NOOBs distribution SD cards. See below in the script section for more details.
Ladies and gentlemen, if you are like me, you probably have a few raspberry pi. (Piis?) I have found them to be great little Linux development boards supported by a huge community. Whenever I need a net connected microprocessor project, the pi is an easy win. I have used it in internet connected clocks, LED signs, RGB lighting projects and so many other crazy projects. They make a great internet gateway for Arduino projects. This use is the very definition of "embedded", a raspberry pi with no keyboard, mouse or monitor. It boots, connects and does it's programmed task. It would be a huge inconvenience to manually configure every one of these starting from a raspbian .img file. Even if your not into electronics, manipulating the SD cards directly will make your life a little easier.
Now would be a great time to introduce my "day job". I am a professional *NIX system administrator and system architect. So when I see a raspberry pi I see a Linux box. UNIX/Linux systems are very easy to clone, it's just a pile of files. I will start with some basic theory and how to work with the SD card from the command line. Then, I will introduce you to my scripts which automate the process.
No Magic hereExactly how the raspberry pi boots is widely misunderstood. And no wonder, it's ummm, complicated. In almost every computer architecture "bootstrap" is messy and complicated. For the user though, the raspberry pi has kept it pretty simple, its just files on the SD. Interface wise, the SD card looks like a disk. Upon that disk are two partitions, a DOS (vfat) partition which contains almost everything necessary for booting, configuration, firmware, and kernel images. Why FAT? Well, when constructing a bootstrap process, the name of the game is simplicity and small code size. The DOS FAT filesystem is very simple compared to modern filesystems, it's also very well understood and pretty universal. The "userland" part of the Linux operating system lives in the second partition which is usually ext4. When the raspberry pi powers on, small chunks of firmware step by step load code into the GPU and CPU until it can eventually load the Linux kernel. This finally makes the system smart enough to mount the slash "/" partition and do all the userland linuxy stuff.So once we understand how it's using the SD card, we can duplicate it and pre-configure if before we even power it on. No magic, It's all just regular files and some *NIX know-how.
Why not just use "dd"? The problem with the standard unix dd utility is it only copies raw disk blocks. For one, your average SD card is mostly unused disk blocks. But dd just copies them sequentially and wastes a lot of time. If you store just the OS files, you get automatic resizing when you restore your archive to a different size SD card. As long as the destination SD is big enough to hold the image, it doesn't matter what size the source card was. Finally, the tar approach allows you to untar and re-tar the archive to modify the archive without it ever booting.
Basic CloningThe first use case we will explore cloning a running raspberry pi to a fresh SD card. To make things easy, we will do all these steps on the actual raspberry pi. You will need to have one free USB port and a USB SD card reader. We will do this step-by-step from the command line, this will give you a good idea how my scripts are constructed later.Power on your Raspberry pi, and log in by whatever means you prefer (graphical, ssh, etc). Get to a command prompt and sudo to a root shell, everything we are going to do needs root privilege. You will also need to install the dosfstools package which is used later. Then, plug in your SD card to any free USB port. Use the "dmesg" utility to dump the kernel log, look at the output to determine what scsi device your new SD card has been assigned. (stuff you type is underlined)
From the above, we can see that our SD card has been assigned "sda" which means /dev/sda is how we will get to it.
First, we need to create the two partitions to hold our /boot and / (slash) filesystems. We do this with the GNU parted utility.
To summarize what we just did. First we use the "mklabel" to create a fresh msdos style partition table on /dev/sda, you will have to answer "y" to the warning. Next we create the partition to hold our FAT /boot filesystem, with a size of 64MB. Next we tell parted to create another partition to hold our slash (/) filesystem starting at the 64MB mark, and ending at the last available sector of the disk (-1s). We tell parted that these two partitions will be fat16 and ext4 respectively so it will set the correct partition type on each.
Finally we "print" the resulting partition table so we can see it, then exit with "quit"
The two partitions we just created are now available as devices /dev/sda1 and /dev/sda2. Next we need to format them. This is done with the mkfs.vfat and mkfs.ext4 commands:
Nice! The output of the "df" command will show us that both slash and boot have been mounted and are the expected sizes. In my case I am using a 8gb SD card. Next, we will execute a command to copy our existing raspbian OS installed on our running pi into our mounted SD card. Now, a couple notes about live cloning. If you are running any services that keep data files open (like mysql, postgresql, apt-get, or other applications) these should be shutdown to insure you get a clean copy. We will use the rsync utility for this copy, we will tell it to only copy / (slash) and /boot, further, we will also tell rsync not to cross filesystem boundaries (--one-file-system).
Why do we specify slash (/) and boot separately? If you look above at the df listing you can see lots of other filesystems mounted. Some of these are ramdisks and virtual filesystems (/sys, /dev, /tmp, and /proc for instance). We do not want to copy these at all. By using the --one-file-system and specifying explicitly slash (/) and /boot, we only capture the OS tree of files. ( Note, the rsync is sensitive to trailing slashes on the source and destination directory names, make sure you run it exactly as below)
Example: Wireless /w DHCP:
Example: Wireless /w static IP:
(NOOBs Users: If you are running a SD card created with NOOBs, you will need to manually modify the boot/cmdline.txt and etc/fstab files on your new SD card before you can boot. You must edit and substitute the mmcblk0p* device names for the proper ones. In the fstab, slash (/) should be /dev/mmcblk0p and /boot should be /dev/mmcblk0p1. In the cmdline.txt file, root= part should be root=/dev/mmcblk0p2. Once you have done that the OS that you backed up will be the only OS on the new SD card event if your NOOBs SD had multiple flavors of raspberry pi OS installed. My scripts on github discussed in the next section handle all of this automatically, which makes things a bit easier.)
These are just a few examples. When your ready to try out your cloned SD card, you should unmount it using the following commands. Make sure you close any shells or change your current working directory (cd) off the new devices or you will get an error when you try to unmount.
You can now safely pull the SD card out and put it in your destination raspberry pi. It should boot and work just like any other SD card.
Sysmatt's ScriptsNow that we have the basic concepts down, Let me show you some of my scripts to automate creating SD cards. These scripts allow you to create and store backup copies of your running raspberry pi systems. You can use this to construct a library of standard, tested, known good images. Very useful when someone wants you to build them a copy of your next talking clock project!We need to retrieve and install the scripts. You will find a link at the end of this article to the current version of the scripts. Download the latest tar archive and untar it into /usr/local/bin as shown below
Download the latest zip from github
Install to /usr/local/bin or any other location you prefer
(NOOBs Users: These scripts have been updated to work with NOOBs SD cards. NOOBs is the SD card image that allows you to have multiple raspberry pi OS installed in different partitions. The way this is handled by the scripts is the OS you are running when you run the backup script is the ONLY os that is backed up. When you run the restore script it will automatically modify the files in /etc and /boot to reflect the changed partition numbers. Essentially this "peels out" one OS from your NOOBs SD and makes it the only OS running on the new (restored) SD. This also automatically grows the restored OS to fill the entire SD card.)
Making a backup archiveIn these scripts I just use a tar archive to store the image. This makes them nicely portable and easy to modify. The first script is used to create the backup copy of the running raspberry pi. This script, "sysmatt.rpi.backup.gtar", by default creates it's backup file under the pi user's home directory with a default filename containing the hostname of the raspberry pi and the datestamp. You can override this filename by specifying an alternate on the command line. Below, I will run this script on the new pi we just cloned.If you so not have enough free disk space on your SD card to hold the tar archive you might want to write it to a flash drive or network mounted filesystem. You should be fine if your SD card is 8gb or larger and have not downloaded too software/data.
To write the tar archive to /home/pi/backups:
Or, to specify an alternate location, run it with a filename argument:
Either of the above will result in a backup copy of the running system being stored in a tar archive file. Again, a note of caution, if you have any programs that are actively writing to data files (mysql/postgresql/apt-get/etc) these should be not run while you make this clone so you get a clean copy.
Restoring / CloningThe next script in the arsenal is a little more complicated. It is called "sysmatt.rpi.restore.sd.card", it automates all the steps we did above with partitioning, creating filesystems, and copying the raspbian image (in the tar archive we just created). This script will destroy everything on the destination SD card, so be careful! First, we insert our destination SD card into a free USB port and discover what /dev/sd? device it has been assigned. You should never assume without checking dmesg!Tip: This script will also work just fine on any Linux box, so you can create SD cards from a workstation also.
Ok, looks like this was assigned /dev/sda also. Don't assume!
Now we run the sysmatt.rpi.restore.sd.card script passing the name of the SD device and the tar archive to restore from. Note, your specific filename will be different.
Above listed is the tar archive we just created. We will now restore it using the next script. Note, it is interactive and will ask you for confirmation. Before the script exits it will give you the opportunity to modify the resulting SD card before it unmounts it.
Now here, the script is paused waiting for input, and you have a choice. If you wish to make changes to the SD card you can open another terminal and modify it under the mountpoint shown (/tmp/pi.sd.324943213 in this example), for instance, change the hostname like we did previously:
When you are done with your modifications, you can return to the session running the script and hit enter. It will unmount the SD card from it's temporary mountpoint. Make sure you have closed any programs or shells that might be holding the filesystems open. You can always try unmounting again with the "umount" command if necessary.
After it is unmounted, you can pull and insert it into your raspberry pi and boot it up. If you want to make the script non-interactive, feel free to remove or comment out the "read" lines that ask for confirmation. But do so at your own risk!
I hope you found this little tutorial helpful. Using these utilities and scripts you can build your own library of standard OS images, preconfigured for particular applications. This makes duplicating projects very easy and quick! If you have feedback on the tutorial or any of the scripts please let me know at matthoskins @ gmail.com or on twitter @sysmatt.