July 4, 2011, 2:57 a.m.
posted by oval
Live Install Your Own Programs
Write your own live installer scripts to install programs you need that aren't on Knoppix by default.
As already briefly discussed in [Hack #27], there are several techniques that allow live installation of software to the home directory. This hack discusses these techniques, shows you how you can create your own live installer modules, and gives an overview of possible future solutions.
1 The Problem
To understand how to write your own live installer scripts, you must understand the main problem with installing software to the home directory: most software expects to be installed in /. Most programs install their files in many different directories in Linux, with files in /usr, /etc, /usr/lib, /var, and other directories. Under Knoppix, the entire /usr directory is part of the read-only CD-ROM filesystem. Even if you copy everything to /home/knoppix/usr, /home/knoppix/etc, /home/knoppix/var, etc., it doesn't work because the program expects its files to be under the root directory. Even if that particular software does not need any other datafiles, most software depends on libraries, and these libraries are also searched for in /usr/lib.
2 Use the Source Luke
The paths that determine where a program looks for its files are set during compile time. And most programs support setting the root to another prefix. With an autotools-compliant package, you can change the prefix by running the following command from the root of the source directory:
[email protected][knoppix]$ ./conFigureprefix=/home/knoppix/.dist/ [email protected][knoppix]$ make [email protected][knoppix]$ make install
Unfortunately, compiling is not a perfect solution. First, compiling a program is slow (depending, of course, on the speed of the computer). If I want to install a program, I want it now, not after a long coffee break. I'm not a Gentoo masochist!
3 Dependency Hell
To install new software in Debian, use the apt-get utility. It not only installs the software you want, but also all dependencies this software needs and all dependencies of the dependencies. Unfortunately, apt-get isn't designed to be run from CD-ROM but from hard disk, and expects certain parts to be writable that are not writable under Knoppix. Despite this problem, the live installer uses wrappers that execute the original programs with special parameters to use apt-get.
4 Patch Me if You Can
After you download the .deb you need, use dpkg to extract the files into your local ~/.dist directory:
[email protected][knoppix]$ dpkg -x file .deb ~/.dist/
To start the program, you must set up a library path so that you can install libraries in places other than the read-only /usr/lib directory. Fortunately, this is possible without the need for root rights. To modify the library path for a program, type:
[email protected][knoppix]$ export PATH=$HOME/.dist/usr/bin:$PATH [email protected][knoppix]$ export LD_LIBRARY_PATH=$HOME/.dist/usr/lib:$LD_LIBRARY_PATH [email protected][knoppix]$ programname
Now the program searches for libraries and programs under your ~/.dist directory—at least most programs work this way. The live installer transparently writes a ~/.bashrc to set these paths so that starting programname in the shell also works. Games that need certain datafiles are still problematic. There isn't a set environment variable for these types of datafiles. Instead, you must determine which files the program is going to reference (dpkg -L packagename at least shows you which files the program includes). This is different for each program, and is one of the challenges that you must face when using live installs. Once you determine the paths that need to be changed, you must path the binary itself to change the hardcoded file paths. For example, tuxracer stores its datafiles in /usr/share/games/tuxracer. To change this path to a local path, run:
[email protected][knoppix]$ perl -pi -e \ 's|/usr/share/games/tuxracer|./../share/games/tuxracer|' $DESTDIR/games/tuxracer
This changes all occurrences of the global path /usr/share/games/tuxracer to the local path /share/games/tuxracer. If you start the program from /home/knoppix/.dist/usr/bin/, it finds the datafiles in /home/knoppix/.dist/usr/share/games/tuxracer.
5 Do It Yourself
OK, now you want to create your own live installer module. The best method is to find one of the existing modules and modify it for your program. To use your own live installer modules, make a directory called modules in your home directory, and run the live installer:
[email protected][knoppix]$ cp -a /usr/share/knx-live-inst/modules ./ [email protected][knoppix]$ cp -a modules/gkrellm modules/myprog [email protected][knoppix]$ knx-live-inst.sh
This command makes a mirror copy of the Knoppix live installer modules in your home directory, and then copies the gkrellm modules to use as a base for your program. The files in modules/myprog allow you to change different settings for your module:
Describes the live installer menu
Is the main executable for the program
Shows which files to download and from where
Explains how to install the program
Is the name of the menu entry for the created Extra Software menu
While description, exes, and menu are simple, the one-line text file, files, has a special format, and install.sh is a shell script.
For a program that can be installed by apt, files is as simple as:
# Format of this is: # basename location_to_download type md5sum myprog none apt none
And install.sh in most cases is as simple as:
#!/bin/bash # # File to actually install the software # copy main prog cp -fa myprog/usr/* $DESTDIR/
Sometimes (such as in the previous tuxracer example), you must add binary patches and wrappers that run when the program is installed to work around difficult file path issues. In these cases, use the tuxracer module as a base so that you can see how it handles patching and wrappers.
6 Advanced Techniques
Some programs, such as the Nvidia drivers, won't work with the previous techniques. The problem is usually that these programs must be in, and write to, /usr. One option is to build a huge symlink farm. In a symlink farm, at boot every directory on the CD-ROM is copied as a new directory in the filesystem, and files in those directories are not copied but symlinked. This makes everything in /usr writable. However, if you build a symlink farm when you create the live CD, you waste space; if you build it while you boot the live CD, it takes a very long time.
One solution to this problem is the dynamic creation of symlink farms. This is how the nvidia live installer works. The installer runs make -n install and checks the output of that program for files that need to be writable, and then builds the necessary symlink farm.
Another possible solution that isn't being implemented in the Knoppix live installer is to use overlay filesystems. The idea behind an overlay filesystem is simple. You can overlay /usr with /ramdisk/usr, and although you can read all files that lie in /usr, all file writes go to /ramdisk/usr. There are several of these filesystems, and they all have some problems:
Is the oldest and best working overlay filesystem (development started with kernel 2.0), but its source code is very complex and not portable to kernel 2.6.
Attempts to redirect files and directories if there is an open or readdir attempt. Translucency unfortunately still lacks support for symlinks, and directory entries that exist in both directories before the overlay are presented twice.
- Linux User Space Filesystem (LUFS)
Is slow because it uses user space tools as opposed to a kernel module.
- Device Mapper
Has a snapshot functionality that doesn't work reliably because it works on the so-called block-layer, which means the underlying filesystem does not know how much space is available.
What all of these solutions have in common is that they don't make use of the VFS Linux Virtual Filesystem Layer (VFS) very much, but reimplement it one way or another. It's possible that eventually a good overlay filesystem will be included in the main kernel tree.