Logo address

A New Grid Server

2015/12/22
2015/12/27 update
2016/02/12 update
2016/02/22 update
2016/03/28 addition

Introduction

Plan9 is designed as a distributed operating system. Therefore, the OS has many nice features for grid computing as shown in the paper by Mirtchovski et al.[1] and also with more detailed descriptions in the thesis by Mirtchovski[2].

Stimulated by their work, we 9fans devoted ourselves to grid computing. From that exercise, we got some fruitful results and ideas to the future. They are summarized in Bell-labs website[3]. However, it seems the activity had ended up without making nice pieces of ideas into reality.

What was wrong? Probably grid computing itself was useless for the people in 9fans. They are interested only in technical problems. It is still open problem what services are useful to the community.

Recently Big Data is talked about. By “Big Data” I mean those data that is inappropriate to be transferred across network. We should note that programs are much smaller then data. Therefore, the data should be processed at the server side by exporting programs to the server. Then we need remote login. In such case, security does matter.

The server I present here is a magical server for those who don't know Plan 9. Clients need to execute their programs in the server, however, the server does not allow any byte to be written into the server's storage. Is it possible to construct such a server that allows execution of the client programs? Yes possible!

Fig.1: Is it possible to accept both requirement?

If we have such a server, we need not to allocate user's storage space in the server, which means the time and labor will be reduced greatly. In addition, we can keep the server perfectly clean.

Ssh is a common tool today for remote execution to unix server. Instead, Plan 9 uses cpu command for remote execution. Cpu command of Plan 9 is different from ssh (or telnet) in that it is not only for command execution but also it mounts local file system to the remote side on the fly as illustrated in Fig.2. The mount point is /mnt/term in the server's namespace.

Fig.2: Plan 9 cpu command mounts local file system to remote side.

Therefore files in local side are visible under /mnt/term in remote side. This means we no longer need traditional tools such as ftp or scp to pass files from locale side to remote side, and furthermore, we need not edit files in remote side. They are editable in local side.

Grid computings today are mainly based on the softwares from Globus[4]. They need distributed accounting with distributed file system so that users are permitted to login and so that storage space is allocated to the users for locating their programs, which however needs high level collaboration. The mechanism illustrated in Fig.2 will make it needless to allocated users space, which means: grid systems may become greatly simplified.

The server I present here is designed dreaming to be a base model for future grid computing that allows freedom for clients to execute their programs, and also that needs perfect cleanness to keep the computing service. The server is constructed on those nice pieces of ideas discussed in plan9 users group[3].

[1] Andrey Mirtchovski, Rob Simmonds and Ron Minnich
Plan 9 – an Integrated Approach to Grid Computing
Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International
http://p9.nyx.link/9grid/plan9-grid.pdf (mirror)

[2] Andrey A. Mirtchovski
Grid Computing with Plan 9 – an Alternative Solution for Grid Computing
http://mirtchovski.com/p9/thesis.pdf (2005)
http:thesis.pdf (mirror)

[3] Plan 9 Wiki — 9grid
http://plan9.bell-labs.com/wiki/plan9/9grid/
http://p9.nyx.link/wiki/9grid/ (mirror)
http://9p.io/wiki/plan9/9grid/index.html (mirror)

[4] Globus
https://www.globus.org/

Login

You need to be registered as a user of plan9.bell-labs.com1.

Execute

cpu -h grid.nyx.link -k 'dom=outside.plan9.bell-labs.com'
with factotum key
key dom=outside.plan9.bell-labs.com proto=p9sk1 user=XXXXX !password=YYYYY
where XXXXX is your ID and YYYYY is your password.

If you succeed in login, you will see a prompt “grid%”.

note 1: If you don't have a user account of outside.plan9.bell-labs.com, please email to me. I can offer you a user account to my grid server. It seems that new bell-labs account is now closed.

The Grid Server

Your name on the server is XXXXX@outside.plan9.bell-labs.com.
Of course the name is not registered as a user to my server.
Thereby you are attached as user none to my file system.
Ramdisk is provided to /tmp which is bound to /usr/none/tmp.

Try

ls /usr
then you will find /usr/none, /usr/arisawa and /usr/OTHERS, where /usr/OTHERS are yours that are produced by
bind -a /mnt/term/usr /usr

You can browse files on the grid server using acme.
However you cannot run commands in acme windows because mount operation is disabled.

The user “XXXXX@outside.plan9.bell-labs.com” has ability to access your local files. Therefore you can edit your files using acme on grid server. (And of course also on your local machine.)

In processing data in grid server, you probably need your own programs. Except a few operations, the 9grid server allows executing your programs even if that are binary executables. Your commands are in /mnt/term/bin.

General speaking, servers can become dirty in multi-users environment. To keep the sanity of server, the 9grid server disables users from writing to permanent storage of the server. If you want to save something, you can write it to your own storage through /mnt/term/usr.

Look /usr/none/lib/profile for the settings and look /usr/arisawa/src/grid for the grid patches.

Local Side Security

If you trust me, you need not read this section.

In accessing 9grid servers, we use cpu command. Then your processes on the 9grid server have ability to access your local machine. This means you have a potential security risk. It is safe to export only a portion of namespace of your file system to 9grid server.

Cpu command has -P option for this purpose. However, unfortunately, this option does not work well.

Another way is to construct minimum namespace for client side.
To do this, create the following files in somewhere, for example in /usr/none/lib.

term% pwd
/usr/none/lib
term% lr -l grid
d-rwxrwxr-x arisawa  sys               0 2015/12/24 13:29:26 grid
d-rwxrwxr-x arisawa  sys               0 2015/12/24 13:26:44 grid/ns
d-rwxrwxr-x arisawa  sys               0 2015/12/16 23:05:16 grid/ns/bin
d-rwxrwxr-x arisawa  sys               0 2015/12/24 12:54:23 grid/ns/dev
--rw-rw-r-- arisawa  sys               0 2015/12/24 12:54:11 grid/ns/dev/cons
--rw-rw-r-- arisawa  sys               0 2015/12/24 12:54:11 grid/ns/dev/consctl
d-rwxrwxr-x arisawa  sys               0 2015/12/24 12:54:23 grid/ns/dev/draw
--rw-rw-r-- arisawa  sys               0 2015/12/24 12:54:11 grid/ns/dev/random
d-rwxrwxr-x arisawa  sys               0 2015/12/25 09:13:14 grid/ns/env
d-rwxrwxr-x arisawa  sys               0 2015/12/24 02:07:26 grid/ns/mnt
d-rwxrwxr-x arisawa  sys               0 2015/12/17 05:41:17 grid/ns/mnt/factotum
--rw-rw-r-- arisawa  sys               0 2015/12/17 05:32:11 grid/ns/mnt/factotum/ctl
--rw-rw-r-- arisawa  sys               0 2015/12/17 05:41:17 grid/ns/mnt/factotum/log
d-rwxrwxr-x arisawa  sys               0 2015/12/24 02:07:26 grid/ns/mnt/wsys
d-rwxrwxr-x arisawa  sys               0 2015/12/16 23:15:39 grid/ns/net
d-rwxrwxr-x arisawa  sys               0 2015/12/24 12:52:41 grid/ns/proc
d-rwxrwxr-x arisawa  sys               0 2015/12/25 09:25:52 grid/ns/usr
d-rwxrwxr-x arisawa  sys               0 2015/12/25 09:25:52 grid/ns/usr/glenda
d-rwxrwxr-x arisawa  sys               0 2015/12/25 09:25:52 grid/ns/usr/none
--rw-rw-r-- arisawa  sys              89 2015/12/24 13:29:45 grid/patt
term%

where glenda is merely an example. and grid/patt is a pattern file for cpu command.

The content is:

- /mnt/factotum
- /mnt/wsys/(.*/)?(text|screen|window)

And have a new cpu command, for example sgcpu (safe guard cpu) with the contents:

#!/bin/rc
rfork ne
cd /usr/none/lib/grid
for(f in cons consctl draw random)
	bind /dev/$f ns/dev/$f
for(f in bin net proc)
	bind /$f ns/$f
bind -c '#e' ns/env
bind -a /mnt/factotum ns/mnt/factotum
bind /mnt/wsys ns/mnt/wsys
bind -a /usr/none ns/usr/none
bind -a /usr/glenda ns/usr/glenda  ## example
bind ns /
cd /usr/none
/bin/cpu -P lib/grid/patt $*

Excute

sgcpu -h grid.nyx.link -k  'dom=outside.plan9.bell-labs.com'

Current Services

Full text database Kirara

grid% kfind
usage: kfind -t database pattern
db is sys, rfc
example:
kfind2 -t rfc esmtp |p
kfind2 -t sys 'exe*n' |p
kfind2 -t sys 'exe*n&fork' |p
kfind2 -t sys 'exe*n|fork' |p
kfind2 -t sys -t rfc 'subm*&smtp' |p
# Two types of wild card, "*" and "?", are allowed.
# In using wild card, they must follow after two or more letters.

you may use databases listed below:
rfc
sources
sys
grid%

Dictionary

grid% kdict
kdict 'query here'  |p  # Eijiro
kdict -d ryaku 'query here'  |p 	# Ryakujiro
kdict -d war 'query here'  |p 	# Tolstoy: War and peace
grid%

The dictionary database is based on Kirara. So same pattern matching rule is applied.

IP address testing

If you want to know your IP address on Internet, then try to access

where luna.nyx.link is the authentication server for grid.nyx.link.
The output format is very simple. The example is
2402:6b00:4040:b600::9!8006
2402:6b00:4040:b600:22c9:d0ff:fe8b:2b5!56645
where the first line is IP address and port number of server, the second line is those of client.

You can access using your web browser with URL

http://grid.nyx.link:8006
or
http://luna.nyx.link:8007
then you can find the IP address which your web browser is using in accessing the URL. Unfortunately you are probably unable to specify the IP address type, IPv6 or IPv4, as far as you are relying on web browsers. Hence we need other ways.

On unix, you can do

telnet [-46] grid.nyx.link 8006
telnet [-46] luna.nyx.link 8007
On Plan9
telnet tcp!grid.nyx.link!8006
telnet tcp!luna.nyx.link!8007
Plan9 telnet cannot control IPv4/IPv6. (use of literal address enable control though)
The command connect is better for this purpose.
connect [-6] tcp!grid.nyx.link!8006
connect [-6] tcp!luna.nyx.link!8007
The default is IPv4.
You can get connect from http://p9.nyx.link/netlib/cmd/connect/ .

The cpu command does not have IPv6 option.
Use literal IPv6 address for the moment, the fix is easy though.

Disabled Operations

If you find a problem, email me please.

Known Bugs

none

Appendix

Technical Notes

2015/12/27

Host owner and user none are special users of Plan9.
Other users are devided in two categories:

Host owner can do everything.
User none is for network services.
Grid users are restricted in small subset of namespace.
For any users, mount/unmount operation is disabled after “rfork m”.

The following operations are disabled for regular users

The following operations are disabled for grid users

Ramdisk is provided for grid users.
Any user (including grid user) can read/write files in client side.

Known problem with process creation is fixed as follows:

This strategy is not perfect, but working reasonably. We need more work to make more robust kernel.

Authentication Mechanism of Plan 9

2016/03/28

A user named alice on Terminal wants to connect to a CPU server named HOST with factotum key:

key proto=p9sk1 dom=DOMa user=IDu !password=PASSu
where IDu is a user name, say alice, on a authentication domain DOMa and PASSu is a password of IDu on DOMa. The command may be
cpu -h HOST -k 'dom=DOMa'
for example.

We assume the CPU server is waiting connections with a factotum key:

key proto=p9sk1 dom=DOMa user=IDa !password=PASSa
where IDa is is a user, say bob, on DOMa and PASSa is a password of IDa on DOMa.

Other possible players might be:

May be IDc != IDr and also may be IDs != IDa and/or IDc != IDu.
The latter case may happen in cross-domain authentication.

Note that: if DOMa is not our domain then we have security problem. For example, IDr may be a user ID for Carol. Therefor cross-domain authentication needs extra special attribute such as grid and the created processes need special owner ID such as IDu@DOMa.

The user IDs must be hostid of the CPU server machine. (look /lib/ndb/auth for hostid.)

In regular situation HOST is a server in domain DOMa. Then the authentication mechanism is well explained in the paper by Russ et al[5].

However we want to consider cross-domain authentication.
For example, HOST is grid.nyx.link and DOMa is outside.bell-labs.com.
Then my question is: how grid.nyx.link can convince that alice is a user in outside.bell-labs.com?

The paper is misleading. Probably the paper is old for understanding cross-domain authentication. Reading the factotum code, you will find that the real processes are essentially:

  1. C → S: CHc
  2. S → C: CHs,DOMa,IDa
  3. C → A: CHs,DOMa,IDa,IDu
  4. A → C: Ku{CHs,IDu,Kn}, Ka{CHs,IDu,Kn}
  5. C → S: Ka{CHs,IDu,Kn}, Kn{CHs,rand}
  6. S → C: Kn{CHc,rand}
where
The essence is:

The set of data (CHs,IDu,Kn) is called ticket.
The value Kn is used for starting SSL encrypted channel.

[5] 􏰒􏰩􏰧􏰧􏰀􏰉􏰤􏰬􏰲􏰀􏰒􏰩􏰧􏰧􏰀􏰉􏰤􏰬􏰲􏰀􏰒􏰩􏰧􏰧􏰀􏰉􏰤􏰬Russ Cox, Eric Crosse, Rob Pike, Dave Presotto and Sean Quilan: Security in Plan 9
http://plan9.bell-labs.com/sys/doc/auth.pdf
http://p9.nyx.link/labs/sys/doc/auth.pdf (mirror)