Logo address

Making grid terminal

目次

Assumption

My system environment before this work

"co" before this work

The host named "co" was one of Plan 9 terminals that had not a disk and was running under a file server named "hera"
The PLAN9.INI was in FDD of "co" and the kernel "9pc" was supplied by hera via network.

Goal

Now I have reconstructed "co" so that it can serve as a grid node. To do this I designed co shown below:

Basic Policy

hostowner

"arisawa" is the most convenient name, because that is my name and my user name of all my host accounts. The followings are owners of our host:

auth server

should be self authenticated, because if users in "co" is authenticated by hera, then they can login to other system such as al. I do want users on "co" can only login to "co".

kernel type

should be 9gpcf, if you want to have ease of mind.
the kernel is based on 9pcf with small patch so that:
I could have 9gpccpuf as well, but 9gpcf is more comfortable, because that is a terminal.

nvram

/dev/sdC0/nvram

password

make new random password for arisawa on "co".
the password will be used for keyfs and factotum.

9gpcf

I might be a paranoia.
If you don't care someone access outside world using your PC, you need not this patch.

/sys/src/9/port/auth.c

prohibit anyone to become "none".

userwrite(char *a, int n)
{
	if(n!=4 || strncmp(a, "none", 4)!=0)
		error(Eperm);

	/* Kenar: only eve can become none */
	if(!iseve())
		error(Eperm);

/sys/src/9/ip/devip.c

prohibit anyone to access outside world.
Fsstdconnect(Conv *c, char *argv[], int argc)
{
	char *p;

	/*	Kenar: only eve and none can connect out */
	if(!iseve() && strcmp(up->user, "none") != 0)
		return "not allowed";

bootable disk

I am on terminal "co" with a new disk on "/dev/sdC0".

mbr

co% disk/mbr -9m /386/mbr /dev/sdC0/data

CAUTION

-9 option initializes /dev/sdC0/data
Manual says:
          The -9
          option initialises the partition table to consist of one
          plan9 partition which spans the entire disc starting at the
          end of the first track.

fdisk

co% disk/fdisk -baw /dev/sdC0/data
# confirmation
co% disk/fdisk /dev/sdC0/data
cylinder = 8225280 bytes
 * p1                     0 10011       (10011 cylinders, 76.68 GB) PLAN9
>>> q
co%

9fat

9fat should be the first of Plan 9 partition.
co% disk/prep /dev/sdC0/plan9
no plan9 partition table found
  empty                  0 160826652   (160826652 sectors, 76.68 GB)
>>> a 9fat 0 40964
>>> w
>>> q
co%

confirmation

co% disk/prep /dev/sdC0/plan9
  9fat                   0 40964       (40964 sectors, 20.00 MB)
  empty              40964 160826652   (160785688 sectors, 76.66 GB)
>>> q
# confirmation
co% ls /dev/sd*
/dev/sdC0/9fat
co%

format

co% cd /386
co% disk/format -b /386/pbslba -d -r 2 /dev/sdC0/9fat 9load 9pcf plan9.ini
add 9load at clust 2
add 9pcf at clust 6c
add plan9.ini at clust 4f8
Initialising FAT file system
type hard, 2 tracks, 255 heads, 63 sectors/track, 512 bytes/sec
Adding file 9load, length 215856
add 9load at clust 2
Adding file 9pcf, length 2381860
add 9pcf at clust 6c
Adding file plan9.ini, length 309
add plan9.ini at clust 4f8
used 2605056 bytes
co%

important

don't forget "-r 2" of prep option.
the first 512B is for MBR and regular partition table of the disk, and I guess, second 512B is for each PBS block for the partition.

confirm by rebooting

Before you reboot you should confirm
you will see a message in success
	MBR...PBS...Plan 9 from Bell Labs
otherwise you will see
	MBR...No active entry
or something else.

If our working is correct until now, you can use "co" as it has been.
Note: that "co" is net booted and file server is hera as it has been.

If not you must use the FD for booting.

partitions for venti and fossil

We continue working on "co" as a terminal with 9pc kernel. The next goal is make spaces for venti and fossil.

prep

The followings are my example.
co% disk/prep /dev/sdC0/plan9
>>> p
  9fat                   0 40964       (40964 sectors, 20.00 MB)
  nvram              40964 40965       (1 sectors, 512 B )
  swap               40965 1040965     (1000000 sectors, 488.28 MB)
  empty            1040965 38798741    (37757776 sectors, 18.00 GB)
  arenas          38798741 122720981   (83922240 sectors, 40.01 GB)
  isect0         122720981 126917093   (4196112 sectors, 2.00 GB)
  empty          126917093 143211310   (16294217 sectors, 7.76 GB)
  fossil         143211310 160826652   (17615342 sectors, 8.39 GB)
>>>
co% ls -l /dev/sd*
--rw-r----- S 0 arisawa arisawa    20973568 Apr  5 10:49 /dev/sdC0/9fat
--rw-r----- S 0 arisawa arisawa 42968186880 Apr  5 10:49 /dev/sdC0/arenas
--rw-r----- S 0 arisawa arisawa           0 Apr  5 10:49 /dev/sdC0/ctl
--rw-r----- S 0 arisawa arisawa 82348277760 Apr  5 10:49 /dev/sdC0/data
--rw-r----- S 0 arisawa arisawa  9019055104 Apr  5 10:49 /dev/sdC0/fossil
--rw-r----- S 0 arisawa arisawa  2148409344 Apr  5 10:49 /dev/sdC0/isect0
--rw-r----- S 0 arisawa arisawa         512 Apr  5 10:49 /dev/sdC0/nvram
--rw-r----- S 0 arisawa arisawa 82343245824 Apr  5 10:49 /dev/sdC0/plan9
-lrw------- S 0 arisawa arisawa           0 Apr  5 10:49 /dev/sdC0/raw
--rw-r----- S 0 arisawa arisawa   512000000 Apr  5 10:49 /dev/sdC0/swap
...
Some comments:

venti

We continue working on "co" as a terminal with 9pc kernel. The next goal is to let venti to work on "co".

before you begin

It is better make dma on before this work, otherwise you will be irritated.
	echo dma on > /dev/sdC0/ctl
It is better to enable dma on. Put the followings somewhere in your termrc,
and mntgen is very convenient.

	for(disk in /dev/sd[CD][01]) {
		if(test -f $disk/ctl)
			echo dma on > $disk/ctl
	}
	mntgen -s slashn && chmod 666 /srv/slashn

NOTE: your termrc might already have a line for mntgen.

fmtisect

co% venti/fmtisect isect0 /dev/sdC0/isect0
clearing the partition
configuring index section isect0 with space for index config bytes=65536
co%

venti.conf

co% cd
co% mkdir venti; cd venti
co% cat>venti.conf <<-EOF-
index	main
isect	/dev/sdC0/isect0
arenas	/dev/sdC0/arenas
-EOF-
co% venti/conf -w /dev/sdC0/arenas venti.conf
co%

fmtarenas

My result is:
co% venti/fmtarenas arena. /dev/sdC0/arenas
clearing the partition
configuring /dev/sdC0/arenas with arenas=80 for a total storage of bytes=42967851008 and directory bytes=65536
adding arena arena.0 at [335872,537206784)
adding arena arena.1 at [537206784,1074077696)
adding arena arena.2 at [1074077696,1610948608)
adding arena arena.3 at [1610948608,2147819520)
adding arena arena.4 at [2147819520,2684690432)
...
adding arena arena.77 at [41339396096,41876267008)
adding arena arena.78 at [41876267008,42413137920)
adding arena arena.79 at [42413137920,42950008832)
co%

venti/fmtindex

co% venti/fmtindex venti.conf
using 262209 buckets of 262216; div=16380
add arena arena.0 at [1048576,537903104)
add arena arena.1 at [537903104,1074757632)
add arena arena.2 at [1074757632,1611612160)
add arena arena.3 at [1611612160,2148466688)
add arena arena.4 at [2148466688,2685321216)
...
add arena arena.77 at [41338847232,41875701760)
add arena arena.78 at [41875701760,42412556288)
add arena arena.79 at [42412556288,42949410816)
configured index=main with arenas=80 and storage=42948362240
co%

starting venti

	venti/venti -w &

confirmation

co% ps
...
arisawa         320    0:00   0:00    2400K Rendez   venti
arisawa         322    0:00   0:00    2400K Rendez   venti
arisawa         323    0:00   0:00    2400K Open     venti
arisawa         325    0:00   0:00     184K Pread    ps
co% netstat -n
...
tcp  3    arisawa    Listen       17034      0          ::
...
term%
you will see venti port 17034

fossil

flfmt

co% fossil/flfmt /dev/sdC0/fossil
co%

conf

flproto1
fsys main config /dev/sdC0/fossil
fsys main open
fsys main
create /active/adm adm sys d775
create /active/adm/users adm sys 664
users -w
srv -p fscons
srv fossil
srv -APW f

writing conf

	fossil/conf -w /dev/sdC0/fossil flproto1

confirmation

co% fossil/conf /dev/sdC0/fossil
fsys main config /dev/sdC0/fossil
fsys main open
fsys main
create /active/adm adm sys d775
create /active/adm/users adm sys 664
users -w
srv -p fscons
srv fossil
srv -APW f
co%
this is only valid at the first time

how to confirm current fossil configuration

Execute:
	fossil/conf /dev/sdC0/fossil

term% fossil/conf  /dev/sdC0/fossil
fsys main config /dev/sdC0/fossil
fsys main open -AWP
fsys main
create /active/adm adm sys d775
create /active/adm/users adm sys 664
users -w
srv -p fscons
srv fossil
term%
note that this configuration is valid only for the beginning

starting fossil

co% venti=co fossil/fossil -f /dev/sdC0/fossil
	nuser 5 len 78
co%
where "co" is my terminal name on which venti is running.

confirming by ps

You will see fossil running in ps outout
co% ps
...
arisawa         284    0:00   0:00    2400K Rendez   venti
arisawa         286    0:17   0:03    2400K Pread    venti
arisawa         287    0:00   0:00    2400K Open     venti
arisawa         291    0:00   0:00    9192K Rendez   fossil
arisawa         292    0:00   0:00    9192K Rendez   fossil
arisawa         293    0:00   0:00    9192K Pread    fossil
arisawa         294    0:02   0:01    2400K Rendez   venti
arisawa         295    0:02   0:22    9192K Rendez   fossil
arisawa         296    0:00   0:00    9192K Sleep    fossil
arisawa         297    0:00   0:00    9192K Rendez   fossil
arisawa         298    7:01   0:00    9192K Rendez   fossil
arisawa         299    0:05   0:01    9192K Pread    fossil
arisawa         300    0:03   0:12    9192K Sleep    fossil
arisawa         301    0:00   0:01    9192K Sleep    fossil
arisawa         302    0:00   0:00    9192K Rendez   fossil
arisawa         303    0:00   0:00    9192K Pread    fossil
arisawa         306    0:00   0:02    9192K Rendez   fossil
arisawa         307    0:01   0:03    9192K Pread    fossil
arisawa         312    0:18   0:03    9192K Rendez   fossil
...

confirming by ls /srv

You will see fossil and fscons and f in /srv.
co% ls /srv
...
/srv/f
/srv/factotum
/srv/fossil
/srv/fscons
/srv/hera
...
where hera is my file server

confirming by mount

before you begin with this you must have /n/f. if not run mntgen
co% 9fs f
srv: /srv/f already exists
mounting as arisawa
co% ls /n/f
/n/f/adm
co% ls /n/f/adm
/n/f/adm/users
co% cat /n/f/adm/users
adm:adm:adm:sys
glenda:glenda:glenda:
none:none::
noworld:noworld::
sys:sys::
co%

Putting files to fossil

cpdir

cpdir is a very convenient tool that enables directory copy. The regular usage is
	cpdir -mvl list source destination
where "list" is a file that describes directory to copy, and source is a source directory destination is a destination directory.

You can get cpdir from:

of course you can use "replica/pull", though.

sources account is required for second case.

basic list file

The minimum is:
386 sys sys
acme sys sys
env sys sys
fd sys sys
lib sys sys
lp sys sys
mail glenda sys
mnt sys sys
n sys sys
rc sys sys
sys sys sys
tmp sys sys
usr sys sys
adm/timezone adm adm
! mail/box/*
! adm/*key*
! adm/whois
! adm/secstore/*

Let assume you have "/srv/f" that is created by

	srv -APW f
in fossil conf

Using "list"

co% 9fs f	# if /n/f is not mounted yet
co% 9fs sources	# we assume you have sources account
post...
co% cpdir -mvl list /n/sources/plan9 /n/f
looking 386 (sys,sys)
creating /n/f/386
creating /n/f/386/9load
..
This takes long time, half day or so.

You might think I could copy existing files on hera, my file server. But consider running system might have private data on it. Safety is my choice.

additional list files

After "list" you can add more to a file "list1" and execute:
	cpdir -mvl list1 /n/sources/plan9 /n/f
list1
68000 sys sys
68020 sys sys
LICENSE sys sys
LICENSE.afpl sys sys
LICENSE.gpl sys sys
NOTICE sys sys
alpha sys sys
amd64 sys sys
arm sys sys
cron sys sys
dist sys sys
lp sys sys
mail sys sys
mips sys sys
power sys sys
sparc sys sys
sparc64 sys sys

/sys/log/*

Look /sys/log. Permission of files in /sys/log should be 666 + a. If not, fix them.
cpdir tries to make permission mode of destination consistent with source, the mode of files in /sources/plan9/sys/log/* are all 664. "replica/pull" will fix this according to plan9.db but cpdir does not.

snap -a

At this stage I recommend you to have an archive
execute:
	con /srv/fscons
and
	snap -a

co% con /srv/fscons
prompt: 	nuser 5 len 78

main: snap -a
main:
snap -a takes long time. It took several tens minutes in my case.
It is safe to wait a message like this one
	archive vac:2e264fb584434715ae845550a87fd520cd4c5ed3
before you write something to new fossil.

Configure your system

/adm/timezone

	cp /n/f/adm/timezone/Japan /n/f/adm/timezone/local

Have your files

Edit /n/f/adm/users
co% cat /n/f/adm/users
adm:adm:adm:sys
arisawa:arisawa:arisawa:
glenda:glenda:glenda:
none:none::
noworld:noworld::
sys:sys::
co%

Create my directory

	mkdir /n/f/usr/arisawa
	mkdir /n/f/usr/arisawa/lib
	mkdir /n/f/usr/arisawa/bin
	mkdir /n/f/usr/arisawa/tmp
And copy some files from your file server for you
For my case
	cpdir -mv /n/hera/usr /n/f/usr arisawa/lib arisawa/bin
Note that I didn't want to copy all my files to "co".
cpu% mkdir /n/f/mail/box/arisawa
cpu% mail -c
creating new mbox: /mail/box/arisawa/mbox
cpu%

/rc/bin/termrc

co% cp /rc/bin/termrc /n/f/rc/bin/termrc
co% cp /lib/ndb/local /n/f/lib/ndb/local

It is better to enable dma on. Put the followings somewhere in your termrc
and mntgen is very convenient.

	for(disk in /dev/sd[CD][01]) {
		if(test -f $disk/ctl)
			echo dma on > $disk/ctl
	}
	mntgen

flproto

Let flproto be
fsys main config /dev/sdC0/fossil
fsys main open
fsys main
users -r /active/adm/users
srv -p fscons
srv fossil
srv -APW f
listen il!*!17008
If you want fossil to be mounted beyond router, add a line
	listen tcp!*!564

Reflect this one to fossil

	fossil/conf -w /dev/sdC0/fossil flproto

PLAN9.INI

add something to your PLAN9.INI
my PLAN9.INI in /n/9fat
[menu]
menuitem=pc
menuitem=pcf
menuitem=gpcf
menudefault=gpcf,60

[pcf]
bootfile=sdC0!9fat!9pcf
bootargs=local!#S/sdC0/fossil
venti=#S/sdC0/arenas
auth=202.250.160.71

[gpcf]
bootfile=sdC0!9fat!9gpcf
bootargs=local!#S/sdC0/fossil
venti=#S/sdC0/arenas
nvram=#S/sdC0/nvram

[pc]
bootfile=ether0!hera!/386/9pc
bootargs=il
fs=202.250.160.71
auth=202.250.160.71

[common]
*nomp=1
ether0=type=rtl8139
monitor=xga
vgasize=1024x768x24
mouseport=ps2intellimouse
#hwaccel=on
hwaccel=off
ipconfig=local
sysname=co

where

	hwaccel=off
	ipconfig=local
is very special to me. My vga card is old and works incorrectly under hwaccel=on. Therefore I put the next code into my termrc.
        if(~ $hwaccel ?*)
                echo hwaccel $hwaccel >/dev/vgactl
ipconfig is also my own variable that is used in my termrc so that I can control IP configuration using PLAN9.INI.
# start ethernet if any; ignore errors
switch($ipconfig){
case local
        eval `{ndb/ipquery sys $sysname ip ipgw ipmask}
        ip/ipconfig -g $ipgw ether /net/ether0 add $ip $ipmask
case dhcp
        ip/ipconfig
case *
        ;
}

CAUTION

menudefault should be gpcf,

preference

/n/f

You need not this if you run mntgen

cpu% mkdir /n/f
mkdir: can't create /n/f: permission denied
cpu% mount /srv/f /n/temp
cpu% mkdir /n/temp/n/f
cpu% unmount /n/temp
cpu% 9fs f
cpu% ls /n/f
/n/f/386
...
cpu%

/usr/local

This is merely my preference.
cp% cpdir -vm /n/hera /n/f usr/local

/lib/namespace.local

This is merely my preference.
# namespace.local follows /lib/namespace
bind -a /usr/local/bin/386 /bin
bind -a /usr/local/bin/rc /bin

halting

	echo sync | con /srv/fscons
	venti/sync
By "venti/sync" you will have a message something like this one:
	packet: 8/132 frag: 9/113 small mem: 11/21 big mem: 3/116

troubles

In "venti/sync"
	fatal error: could not connect to server: cs: can't translate service
Probably you don't have $venti, retry:
	venti=co venti/sync
where "co" is the host on which venti is running.

To be an authentication server

Registering authentication data

cpu% ls -l /adm
--rw-rw-r-- M 102 arisawa sys      81 Jun  3 15:19 /adm/keys
--rw-rw-r-- M 102 arisawa sys      46 Jun  3 15:20 /adm/keys.who
d-rwxrwxr-x M 102 adm     adm       0 May 22 13:46 /adm/timezone
--rw-rw-r-- M 102 adm     arisawa 103 Jun  3 12:41 /adm/users
--rw-rw-r-- M 102 arisawa sys       0 Jun  3 15:15 /adm/whois
cpu% auth/keyfs -wp -m /mnt/keys /adm/keys >/dev/null >[2=1]
cpu% auth/changeuser arisawa
Password:
Confirm password:
assign Inferno/POP secret? (y/n) n
Expiration date (YYYYMMDD or never)[return = never]:
Post id:
User's full name: Kenji Arisawa
Department #:
User's email address: arisawa@aichi-u.ac.jp
Sponsor's email address:
user arisawa installed for Plan 9
cpu%


cpu% touch /n/f/adm/keys
cpu% touch /n/f/adm/keys.who
cpu% ls -l /adm
--rw-rw-r-- M 102 arisawa sys   0 Jun  3 15:11 /adm/keys
--rw-rw-r-- M 102 arisawa sys   0 Jun  3 15:12 /adm/keys.who
d-rwxrwxr-x M 102 adm     adm   0 May 22 13:46 /adm/timezone
--rw-rw-r-- M 102 adm     sys 103 Jun  3 12:41 /adm/users
cpu%

make nvram

nvram data is used by auth/keyfs in reading /adm/keys

cpu% auth/wrkey
bad nvram key
bad authentication id
bad authentication domain
authid: arisawa
authdom: co.aichi-u.ac.jp
secstore key: XXXXXXXXXX
password: XXXXXXXXXX
cpu%

PLAN9.INI

Add a line

nvram='#S/sdC0/nvram'

/rc/bin/service

strip out unnecessary services.
We use only
	tcp17007
	tcp17009
	tcp17010
	tcp17013

/rc/bin/service.auth

We use
	il566
	tcp567

/rc/bin/termrc

key/fs in termrc
Change the service

last portion of /rc/bin/termrc

...
# auth/keyfs -wp -m /mnt/keys /adm/keys >/dev/null >[2=1]
auth/keyfs -wp -m /mnt/keys /adm/keys >/dev/null >[2=1]
# cron job might be used to monitor something such as endless jobs
# auth/cron >>/sys/log/cron >[2=1] &
auth/cron >>/sys/log/cron >[2=1] &
# co is a grid node owned by arisawa
# the users are authenticated by this terminal.
aux/listen -q -t /rc/bin/service.auth -d /rc/bin/service il
aux/listen -q -t /rc/bin/service.auth -d /rc/bin/service tcp

/lib/ndb/local

#
#  co: Plan9 grid node
#
auth=co.aichi-u.ac.jp authdom=co.aichi-u.ac.jp
...
ip=202.250.160.139 ether=0020ed7a45cc sys=co
	dom=co.aichi-u.ac.jp
	bootf=/386/9gpcf
	proto=il
	auth=co
	fs=co
I am not sure if this is OK.

/lib/ndb/auth

hostid=arisawa
	uid=!sys uid=!adm uid=*

reboot again

confirmation

some protection

confirm the effect of kernel 9gpcf
co% su alice
su# su
su: can't become none
su# telnet vega
telnet: not allowed
su#
These are kernel lebel protection.

Who can login ?

cat /mnt/factotum/ctl will looks like
co% fact
--rw-r--r-- M 138 arisawa arisawa 0 Jun  3 17:03 /mnt/factotum/ctl
key proto=p9sk1 dom=aichi-u.ac.jp user=arisawa !password?
key proto=p9sk1 dom=outside.plan9.bell-labs.com user=arisawa !password?


term% cpu -h co -u bob

!Adding key: dom=aichi-u.ac.jp proto=p9sk1 user=bob
password:
!
% me
bob             427    0:00   0:00     184K Pread    ps
--rw-rw-rw- M 159 none sys 0 Jun  3 17:03 /usr/none/tmp/me
%

CAUTION

factotum of bootes in hera (our file server ) should be
key proto=p9sk1 user=bootes dom=aichi-u.ac.jp !hex? !password?
key proto=p9sk1 dom=outside.plan9.bell-labs.com role=client user=arisawa !password?
That is, you must specify "role=client" for bell-labs key,
otherwise hera is accesible by any sources users.
Likewise

Adding some users for "co"

co% auth/changeuser alice
Password:
Confirm password:
assign Inferno/POP secret? (y/n) n
Expiration date (YYYYMMDD or never)[return = never]:
changeuser: can't create user g1: permission denied
co%
This message can come when you are working from remote machine.
In that case you cannot see /mnt/keys.
try
	auth/keyfs
and then
	auth/changeuser

Troubles

'tmp' file does not exist

term% cpu -h co
bind: tmp: 'tmp' file does not exist
cpu% me
arisawa         395    0:00   0:00     184K Pread    ps
--rw-rw-rw- M 97 arisawa sys 0 Jun  3 17:59 /usr/none/tmp/me
cpu% ls /tmp
/tmp/A343.arisacme
/tmp/list3
'/tmp/ts.co.2.$ntp'
/tmp/xxx
cpu%
This might came from a line in $home/lib/profile
	bind -c tmp /tmp
Change to
	bind -c $home/tmp /tmp

More refinement

using secstore server

I want to use hera as secstore server of arisawa.

Solution:
	auth/secstore -s hera ...

Security Consideration

factotum

my factotum of host owner of co is
key proto=p9sk1 dom=co.aichi-u.ac.jp user=arisawa !password=ABC
key proto=p9sk1 dom=outside.plan9.bell-labs.com user=arisawa !password=DEF
key proto=p9sk1 dom=grid.bell-labs.com user=arisawa !password=XYZ
where ABC,DEF and XYZ are password.