Count from April/13/2005

HTTP-FUSE-KNOPPIX / Xenoppix

[CAUTION] The home pages related to KNOPPIX Japanese Edition, VMKnoppix, etc moved to http://www.rcis.aist.go.jp/project/knoppix/index-en.html
Please visit the new home page for latest information.

Japanese


Special Topic:


Introduction

HTTP-FUSE-KNOPPIX is a "pile-up" KNOPPIX. HTTP-FUSE KNOPPIX downloads pieces of Root Filesystem from a HTTP server when the piece is requested, although normal KNOPPIX requires whole 700MB ISO image at first. The piece is "piled up" on your PC.
The first minimum parts of HTTP-FUSE-KNOPPIX are bootloader with "Linux kernel" and "miniroot". The size is only 6MB. The rest of parts "ROOT file system" which size is 700MB is downloaded on demand as a small piece of block device.
The original block device which includes ROOT file system is split into small data-piece. Each small data-piece is compressed and saved to a file. The files are called "split-and-compressed block files". Split-and-compressed block files is downloaded when it is required. Split-and-compressed block files compose a virtual block device with HTTP-FUSE..
# HTTP-FUSE is based on FUSE (Filesystem in Userspace) http://fuse.sourceforge.net/.

HTTP-FUSE-KNOPPIX selects ROOT file system at boot time. It means HTTP-FUSE-KNOPPIX doesn't make a CD-ROM for customization. It just upload the customized Root File system on HTTP server.
Furthermore the uploaded "split-and-compressed block files" for customized KNOPPIX are difference only. The most split-and-compressed blocks files are shared between original KNOPPIX and customized KNOPPIX. The feature makes small volume of a server when customized KNOPPIX is added.

The following figures show the difference of "Normal KNOPPIX" and "HTTP-FUSE-KNOPPIX".

Normal KNOPPIX

ISO image is 700MB.
It includes everything (bootloader, kernel, miniroot, and cloop file which includes ROOT file system).


HTTP-FUSE-KNOPPIX

The fist boot image is 6MB. It includes bootloader, kernel, and miniroot.
ROOT File System (cloop file) is changed to "split-and-compressed block files".
The files are obtained form HTTP server via HTTP-FUSE. and re-composed

The pieces of cloop file is plied to local storage. They are re-usable.

ROOT File System is selectable at boot time. We don't need to burn CD-ROM for customized KNOPPIX.

The same block files are shared between original and customized KNOPPIX.


How to boot first

The boot time of HTTP-FUSE-KNOPPIX deeply depends on network latency.


We use CDN(Contents Delivery Network) to distribute block files. The current target CDNs are RING Project Mirror servers and "Coralised" servers ("Coral" is project for P2P proxies) . The CDNs have many sites and allocate near (short latency) sites for a client by DNS in the CDN. Many Ring servers exist in Japan and offers broad-band network for Japanese clients. Coral Proxies are distributed by PlanetLab and covers world wide.

Furthermore HTTP-FUSE-KNOPPIX use "netselect" to find a nearest site among CDNs. It enables us to get shortest network latency at anywhere in the world.

"netselect" finds the closest site on CDN.


The summary of benefits

------------------------------------
Learn a bitter lesson from
SFS-KNOPPIX


Before HTTP-FUSE-KNOPPIX, we made SFS-KNOPPIX which boots form Internet with a cloop (ROOT file system) on SFS(Self-certifying File System). We learn a bitter lesson form SFS-KNOPPIX. SFS servers exists in Japan. There is long network latency (more than 100msec) form US and EU. It causes 30 MINUTES to boot SFS-KNOPPIX from US and EU.

The problem is caused by Server-Client model on Internet. SFS-KNOPPIX have to connect SFS to mount ROOT file system every boot and can't escape form latency. To solve the problem we make HTTP-FUSE-KNOPPIX. The points are the following.


Download

HTTP-FUSE-KNOPPIX(CD Version)

HTTP-FUSE-KNOPPIX(USB Version)


Requirement


Preparation

HTTP-FUSE-KNOPPIX(CD Version)

HTTP-FUSE-KNOPPIX(USB Version)

HTTP-FUSE-KNOPPIX USB version can install from WindowsXP/2000.


To save block files


BOOT


Special Boot Option


HTTP Servers & Block files

List of HTTP Servers: http://unit.aist.go.jp/itri/knoppix/knxblock/deliver_menu


List of blocks files: http://unit.aist.go.jp/itri/knoppix/knxblock/contents_menu

"*.cue" file is a list of block files. You can check the difference with "diff" command.


Detail of Implementation

The figure shows the detail of cloop driver. A cloop file is made of a block device. Block Device is split to 64KB and compressed. They are stored to a cloop file. The header of cloop file has location data of split-and-compressed blocks. Cloop driver re-configures a virtual block device with cloop file when it is accessed.


The figure shows the detail of HTTP-FUSE and cloop driver. Block Device is split to 64KB and compressed Each piece is save as a file. We call it "split-and-compressed block file".HTTP-FUSE gets split-and-compressed block files and re-configures a coop files.




The figure shows HTTP-FUSE driver gets split-and-compressed block files from HTTP server




Reduce the block files for customized KNOPPIX, because the most block files aren't changed and reusable. The following figure shows how to get the difference block files.


HTTP-FUSE mount & mkmd5files

(OLD) HTTP-FUSE mount & mkmd5files


Problems and Solutions


Reference

Related work


Acknowledgement

A part of this development is supported by Exploratory Software Project of IPA (Information Technology Promotion Agency, Japan). We would like to thank Fumitoshi Ukai(Project Manager) and Jun Okajima(Co-developer) for idea and implementation.