riscv-simulator

发表于 | 分类于 | 阅读次数：
本文字数： 38k | 阅读时长 ≈ 35 分钟

riscv simulator

Base environment

Debian

Install Debian and Terminalin Microsoft Store. With this app you get Debian for the Windows Subsystem for Linux (WSL). You will be able to use a complete Debian command line environment containing a full current stable release environment.

Modify the source list.

1
2
3
4
5
6
7
8
9
10
$ sudo vi /etc/apt/sources.list
deb http://mirrors.163.com/debian/ buster main non-free contrib
deb http://mirrors.163.com/debian/ buster-updates main non-free contrib
deb http://mirrors.163.com/debian/ buster-backports main non-free contrib
deb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib

deb-src http://mirrors.163.com/debian/ buster main non-free contrib
deb-src http://mirrors.163.com/debian/ buster-updates main non-free contrib
deb-src http://mirrors.163.com/debian/ buster-backports main non-free contrib
deb-src http://mirrors.163.com/debian-security/ buster/updates main non-free contrib

Install some library.

1
2
3
4
5
6
$ sudo apt update
$ sudo apt install vim
$ sudo apt install tree
$ sudo apt install ripgrep
$ sudo apt install git
$ sudo apt install zsh

RISCV toolchain

GNU

You can compile riscv toolchain by yourself, but it is not easy and it often has strange issue.

So I highly recommend using the distro.

1
sudo apt-get install gcc-riscv64-linux-gnu

If you still want to compile the toolchain, you can refer to below commands.

1
2
$ git clone --recursive https://github.com/riscv/riscv-gnu-toolchain
$ sudo apt-get install autoconf automake autotools-dev curl python3 libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev libnewlib-dev

To build the Newlib cross-compiler, pick an install path. If you choose, say, /opt/riscv, then add /opt/riscv/bin to your PATH now. Then, simply run the following command:

1
2
$ ./configure --prefix=/home/francis/opt/riscv
$ make -j8

The riscv64-unknown-elf-gcc will be installed. The C runtime library used is newlib.

To build the Linux cross-compiler, pick an install path. If you choose, say, /opt/riscv, then add /opt/riscv/bin to your PATH now. Then, simply run the following command:

1
2
$ ./configure --prefix=/home/francis/opt/riscv
$ make linux -j8

The riscv64-unknown-linux-gcc will be installed. The C runtime library used is glibc.

Finally, add path to shrc.

1
2
create RISCV /home/francis/opt/riscv
insert PATH $RISCV/bin

Spike

Spike, the RISC-V ISA Simulator, implements a functional model of one or more RISC-V harts. It is named after the golden spike used to celebrate the completion of the US transcontinental railway.

1
2
3
4
5
6
7
$ git clone https://github.com/riscv/riscv-isa-sim.git
$ apt-get install device-tree-compiler
$ mkdir build
$ cd build
$ ../configure --prefix=$RISCV
$ make
$ make install

The above command will create $RISCV/bin/spike.

riscv-pk

The RISC-V Proxy Kernel, pk, is a lightweight application execution environment that can host statically-linked RISC-V ELF binaries. It is designed to support tethered RISC-V implementations with limited I/O capability and thus handles I/O-related system calls by proxying them to a host computer.

This package also contains the Berkeley Boot Loader, bbl, which is a supervisor execution environment for tethered RISC-V systems. It is designed to host the RISC-V Linux port.

1
2
3
4
5
6
$ git clone https://github.com/riscv/riscv-pk.git
$ mkdir build
$ cd build
$ ../configure --prefix=$RISCV --host=riscv64-unknown-elf
$ make
$ make install

The above command will create $RISCV/riscv64-unknown-elf/pk and bbl.

hello world on spike

If this test works, it means your tool chain is correctly installed.

1
2
3
4
5
$ echo -e '#include <stdio.h>\n int main(void) { printf("Hello world!\\n"); return 0; }' > hello.c
$ riscv64-unknown-elf-gcc -o hello hello.c
$ spike /home/francis/opt/riscv/riscv64-unknown-elf/bin/pk hello
bbl loader
Hello world!

qemu

There are two version qemu. one is master, it is master version. The other is kvm_qemu, it support the hypervisor v0.6.

If you want master version, you can use this command.

1
$ git clone https://github.com/qemu/qemu.git

If you want kvm version, you can use this command.

1
$ git clone https://github.com/kvm-riscv/qemu.git

Install command.

1
2
3
4
$ sudo apt-get install gcc libc6-dev pkg-config bridge-utils uml-utilities zlib1g-dev libglib2.0-dev autoconf automake libtool libsdl1.2-dev

$ ./configure --target-list=riscv64-softmmu
$ make

The above command will create riscv64-softmmu/qemu-system-riscv64

Run Kernel on spike

Download linux.

1
$ git clone https://github.com/torvalds/linux.git

busy box

Download busybox.

1
2
3
4
5
$ curl -L http://busybox.net/downloads/busybox-1.26.2.tar.bz2 >busybox-1.26.2.tar.bz2
$ tar xvjf busybox-1.26.2.tar.bz2
$ cd busybox-1.26.2
$ make allnoconfig  
$ make menuconfig

Enter config menu, and open below configs.

1
2
3
4
5
6
7
8
9
10
11
12
CONFIG_STATIC=y, listed as “Build BusyBox as astatic binary (no shared libs)” in BusyBox Settings Build Options
CONFIG_FEATURE_INSTALLER=y, listed as“Support –install [-s] toinstall applet links at runtime” in BusyBox Settings General Configuration
CONFIG_CROSS_COMPILER_PREFIX=riscv64-unknown-linux-gnu-,listed as “Cross Compiler prefix” inBusyBox Settings  Build Options
CONFIG_INIT=y, listed as “init” in Init utilities
CONFIG_ASH=y, listed as “ash” in Shells
CONFIG_ASH_JOB_CONTROL=n, listed as “Ash → Job control” inShells
CONFIG_MOUNT=y, listed as “mount” in Linux SystemUtilities
CONFIG_FEATURE_USE_INITTAB=y, listed as “Support reading aninittab file” in Init Utilities
CONFIG_ASH_BUILTIN_ECHO=y, listed as “echo” in Shells
CONFIG_ASH_BUILTIN_PRINTF=y, listed as “printf” in Shells
CONFIG_ASH_BUILTIN_TEST=y, listed as “test” in Shells
CONFIG_ASH_HELP=y, listed as “help” in Shells
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ cd ..
$ vim inittab
::sysinit:/bin/busybox mount -t proc proc /proc
::sysinit:/bin/busybox mount -t tmpfs tmpfs /tmp
::sysinit:/bin/busybox --install -s
/dev/console::sysinit:-/bin/ash 

$ vim make_rootfs.sh
CDIR=$PWD
if [ -z "$BUSYBOX" ]; then BUSYBOX=$CDIR/busybox-1.26.2; fi

if [ -z "$LINUX" ]; then LINUX=$CDIR/linux; fi

if [ -d "$BUSYBOX" ] && [ -d "$LINUX" ]; then
    echo "build busybox..."
    make -j$(nproc) -C "$BUSYBOX" 2>&1 1>/dev/null
    if [ -d rootfs ]; then rm -fr rootfs; fi
    mkdir rootfs && 
    cd rootfs
    mkdir -p  bin etc dev lib proc sbin sys tmp usr usr/bin usr/lib usr/sbin
    cp "$BUSYBOX"/busybox bin/
    ln -s bin/busybox ./init
    cp $CDIR/inittab etc/inittab
    echo "\
        mknod dev/console c 5 1 && \
        find . | cpio -H newc -o > "$LINUX"/rootfs.cpio\
        " | fakeroot &&
    if [ $? -ne 0 ]; then echo "build busybox failed!"; fi
else
    echo "make sure you have both linux and busybox downloaded."
fi

sudo ./make_rootfs.sh

The above command will create ../linux/rootfs.cpio

kernel

1
2
3
4
$ cd linux
$ cp syven.config .config
$ make -j8 ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- olddefconfig
$ make -j8 ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu-

The above command will create ../linux/rootfs.cpio

You also can use below command to configure kernel.

1
2
3
4
5
6
$ make -j8 ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- defconfig
$ sudo apt-get install  ncurses-dev
$ make menuconfig
CONFIG_VT_CONSOLE=n
Initramfs source file 输入Enter键将其更改为“<riscv_linux>/rootfs.cpio”
CONFIG_HVC_RISCV_SBI=y

pk

Enter riscv-pk directory, package vmlinux to bbl。

1
2
3
4
5
$ cd <riscv-pk>/build
$ rm -rf *
$ ../configure  --prefix=$RISCV --host=riscv64-linux-gnu --with-payload=<riscv-linux>/vmlinux
$ make
$ make install

run kernel

1
$ spike bbl

Run KVM on qemu

Currently, we can boot RISC-V 64bit SMP Guest using KVM RISC-V on QEMU. We use QEMU as our development vehicle for RISC-V hypervisor development.

To achieve this, we need following components:

  1. QEMU with RISC-V Hypervisor Extension Emulation
  2. OpenSBI Firmware with Hypervisor Extension Support
  3. Common Host & Guest Kernel
  4. KVMTOOL
  5. Host RootFS with KVMTOOL and Guest Kernel

1. Build QEMU with RISC-V Hypervisor Extension Emulation

Note: You must download kvm-riscv/qemu, NOT qemu master. 

1
2
3
4
5
git clone https://github.com/kvm-riscv/qemu.git
cd qemu
./configure --target-list="riscv32-softmmu riscv64-softmmu"
make
cd ..

The above commands will create ./qemu/riscv64-softmmu/qemu-system-riscv64 which will be our QEMU system emulator.

2. Build OpenSBI Firmware with Hypervisor Extension Support

1
2
3
4
git clone https://github.com/riscv/opensbi.git
cd opensbi
make CROSS_COMPILE=riscv64-linux-gnu- PLATFORM=generic
cd ..

The above commands will create opensbi/build/platform/generic/firmware/fw_jump.bin which will be our M-mode runtime firmware.

3. Build Common Host & Guest Linux Kernel Image

We can use same RISC-V 64bit Linux kernel as Guest and Host kernel so no need to compile them separately.

Note: You must download kvm-riscv/linux, NOT linux master. 

1
2
3
4
git clone https://github.com/kvm-riscv/linux.git
mkdir build-riscv64
make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -C linux O=`pwd`/build-riscv64 ARCH=riscv defconfig
make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -C linux O=`pwd`/build-riscv64 ARCH=riscv

The above commands will create build-riscv64/arch/riscv/boot/Image which will be our Guest and Host kernel.

4. Add libfdt library to CROSS_COMPILE SYSROOT directory

We need libfdt library in the cross-compile toolchain for compiling KVMTOOL RISC-V (described in next step). The libfdt library is generally not available in the cross-compile toolchain so we need to explicitly compile libfdt from DTC project and add it to CROSS_COMPILE SYSROOT directory.

1
2
3
4
5
6
7
8
git clone git://git.kernel.org/pub/scm/utils/dtc/dtc.git
cd dtc
export CC="${CROSS_COMPILE}gcc -mabi=lp64d -march=rv64gc"
TRIPLET=$($CC -dumpmachine)
SYSROOT=$($CC -print-sysroot)
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- libfdt
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- EXTRA_CFLAGS="-mabi=lp64d" DESTDIR=$SYSROOT PREFIX=/usr LIBDIR=/usr/lib64/lp64d install-lib install-includes
cd ..

The above commands will install cross-compiled libfdt library at $SYSROOT/usr/lib64/lp64d directory of cross-compile toolchain.

5. Build KVMTOOL

1
2
3
4
5
git clone https://github.com/kvm-riscv/kvmtool.git
cd kvmtool
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- lkvm-static
riscv64-linux-gnu-strip lkvm-static
cd ..

The above commands will create kvmtool/lkvm-static which will be our user-space tool for KVM RISC-V.

6. Build Host RootFS containing KVMTOOL and Guest Linux

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
git clone https://github.com/kvm-riscv/howto.git
wget https://busybox.net/downloads/busybox-1.27.2.tar.bz2
tar -C . -xvf ./busybox-1.27.2.tar.bz2
mv ./busybox-1.27.2 ./busybox-1.27.2-kvm-riscv64
cp -f ./howto/configs/busybox-1.27.2_defconfig busybox-1.27.2-kvm-riscv64/.config
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -C busybox-1.27.2-kvm-riscv64 oldconfig
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -C busybox-1.27.2-kvm-riscv64 install
mkdir -p busybox-1.27.2-kvm-riscv64/_install/etc/init.d
mkdir -p busybox-1.27.2-kvm-riscv64/_install/dev
mkdir -p busybox-1.27.2-kvm-riscv64/_install/proc
mkdir -p busybox-1.27.2-kvm-riscv64/_install/sys
mkdir -p busybox-1.27.2-kvm-riscv64/_install/apps
ln -sf /sbin/init busybox-1.27.2-kvm-riscv64/_install/init
cp -f ./howto/configs/busybox/fstab busybox-1.27.2-kvm-riscv64/_install/etc/fstab
cp -f ./howto/configs/busybox/rcS busybox-1.27.2-kvm-riscv64/_install/etc/init.d/rcS
cp -f ./howto/configs/busybox/motd busybox-1.27.2-kvm-riscv64/_install/etc/motd
cp -f ./kvmtool/lkvm-static busybox-1.27.2-kvm-riscv64/_install/apps
cp -f ./build-riscv64/arch/riscv/boot/Image busybox-1.27.2-kvm-riscv64/_install/apps
cd busybox-1.27.2-kvm-riscv64/_install; find ./ | cpio -o -H newc > ../../rootfs_kvm_riscv64.img; cd -

The above commands will create rootfs_kvm_riscv64.img which will be our Host RootFS containing KVMTOOL and Guest Linux.

7. Run RISC-V KVM on QEMU

Run Host Linux with Host RootFS on QEMU

1
./qemu/riscv64-softmmu/qemu-system-riscv64 -monitor null -cpu rv64,x-h=true -M virt -m 512M -display none -serial mon:stdio -bios opensbi/build/platform/generic/firmware/fw_jump.bin -kernel ./build-riscv64/arch/riscv/boot/Image -initrd ./rootfs_kvm_riscv64.img -append "root=/dev/ram rw console=ttyS0 earlycon=sbi"

Run Guest Linux using KVMTOOL after Host Linux is booted

1
./apps/lkvm-static run -m 128 -c2 --console serial -p "console=ttyS0 earlycon=uart8250,mmio,0x3f8" -k ./apps/Image --debug

8. Reference Bootlog

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
anup@anup-ubuntu64:~/Work/riscv-test/kvm-riscv$ ./qemu/riscv64-softmmu/qemu-system-riscv64 -monitor null -cpu rv64,x-h=true -M virt -m 512M -display none -serial mon:stdio -bios opensbi/build/platform/generic/firmware/fw_jump.bin -kernel ./build-riscv64/arch/riscv/boot/Image -initrd ./rootfs_kvm_riscv64.img -append "root=/dev/ram rw console=ttyS0 earlycon=sbi"

OpenSBI v0.4-33-g118499a (Sep  3 2019 10:28:17)
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name          : QEMU Virt Machine
Platform HART Features : RV64ACDFHIMSU
Platform Max HARTs     : 8
Current Hart           : 0
Firmware Base          : 0x80000000
Firmware Size          : 112 KB
Runtime SBI Version    : 0.1

PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0xffffffffffffffff (A,R,W,X)
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.3.0-rc6-00027-g9731700e1faf (anup@anup-ubuntu64) (gcc version 8.2.0 (Buildroot 2018.11-rc2-00003-ga0787e9)) #1 SMP Tue Sep 3 10:35:02 IST 2019
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] Initial ramdisk at: 0x(____ptrval____) (12421632 bytes)
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000] software IO TLB: mapped [mem 0x9b8fb000-0x9f8fb000] (64MB)
[    0.000000] riscv: ISA extensions acdfhimsu
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] percpu: Embedded 18 pages/cpu s34776 r8192 d30760 u73728
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 128775
[    0.000000] Kernel command line: root=/dev/ram rw console=ttyS0 earlycon=sbi
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 427292K/522240K available (6061K kernel code, 381K rwdata, 1886K rodata, 213K init, 305K bss, 94948K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] plic: mapped 53 interrupts with 1 handlers for 2 contexts.
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
[    0.000208] sched_clock: 64 bits at 10MHz, resolution 100ns, wraps every 4398046511100ns
[    0.006360] Console: colour dummy device 80x25
[    0.013724] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000)
[    0.015615] pid_max: default: 32768 minimum: 301
[    0.017640] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.018786] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.086621] rcu: Hierarchical SRCU implementation.
[    0.096994] smp: Bringing up secondary CPUs ...
[    0.097897] smp: Brought up 1 node, 1 CPU
[    0.166360] devtmpfs: initialized
[    0.179615] random: get_random_u32 called from bucket_table_alloc.isra.10+0x4e/0x160 with crng_init=0
[    0.184872] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.186214] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.197244] NET: Registered protocol family 16
[    0.357230] vgaarb: loaded
[    0.365369] SCSI subsystem initialized
[    0.378003] usbcore: registered new interface driver usbfs
[    0.380074] usbcore: registered new interface driver hub
[    0.382201] usbcore: registered new device driver usb
[    0.399505] clocksource: Switched to clocksource riscv_clocksource
[    0.538060] NET: Registered protocol family 2
[    0.548385] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.550609] TCP established hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.551592] TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.561878] TCP: Hash tables configured (established 4096 bind 4096)
[    0.564518] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.566006] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.573721] NET: Registered protocol family 1
[    0.583756] RPC: Registered named UNIX socket transport module.
[    0.584443] RPC: Registered udp transport module.
[    0.584854] RPC: Registered tcp transport module.
[    0.585318] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.585971] PCI: CLS 0 bytes, default 64
[    0.591419] Unpacking initramfs...
[    0.855083] Freeing initrd memory: 12128K
[    0.856104] kvm [1]: hypervisor extension available
[    0.856811] kvm [1]: host has 14 VMID bits
[    0.861361] workingset: timestamp_bits=62 max_order=17 bucket_order=0
[    0.921884] NFS: Registering the id_resolver key type
[    0.925385] Key type id_resolver registered
[    0.927104] Key type id_legacy registered
[    0.928231] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.933015] 9p: Installing v9fs 9p2000 file system support
[    0.937670] NET: Registered protocol family 38
[    0.939524] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.942659] io scheduler mq-deadline registered
[    0.945381] io scheduler kyber registered
[    0.952696] pci-host-generic 30000000.pci: host bridge /soc/pci@30000000 ranges:
[    0.954141] pci-host-generic 30000000.pci:    IO 0x03000000..0x0300ffff -> 0x00000000
[    0.956787] pci-host-generic 30000000.pci:   MEM 0x40000000..0x7fffffff -> 0x40000000
[    0.967796] pci-host-generic 30000000.pci: ECAM at [mem 0x30000000-0x3fffffff] for [bus 00-ff]
[    0.971617] pci-host-generic 30000000.pci: PCI host bridge to bus 0000:00
[    0.973162] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.974141] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.976144] pci_bus 0000:00: root bus resource [mem 0x40000000-0x7fffffff]
[    0.979901] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[    1.270893] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.278968] printk: console [ttyS0] disabled
[    1.280820] 10000000.uart: ttyS0 at MMIO 0x10000000 (irq = 10, base_baud = 230400) is a 16550A
[    1.286134] printk: console [ttyS0] enabled
[    1.286134] printk: console [ttyS0] enabled
[    1.288244] printk: bootconsole [sbi0] disabled
[    1.288244] printk: bootconsole [sbi0] disabled
[    1.300170] [drm] radeon kernel modesetting enabled.
[    1.360503] loop: module loaded
[    1.367832] libphy: Fixed MDIO Bus: probed
[    1.370085] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.371286] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.372665] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.373698] ehci-pci: EHCI PCI platform driver
[    1.375530] ehci-platform: EHCI generic platform driver
[    1.376687] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.377491] ohci-pci: OHCI PCI platform driver
[    1.378782] ohci-platform: OHCI generic platform driver
[    1.380448] usbcore: registered new interface driver uas
[    1.381446] usbcore: registered new interface driver usb-storage
[    1.384452] mousedev: PS/2 mouse device common for all mice
[    1.387828] usbcore: registered new interface driver usbhid
[    1.388569] usbhid: USB HID core driver
[    1.392400] NET: Registered protocol family 10
[    1.407744] Segment Routing with IPv6
[    1.409678] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    1.417929] NET: Registered protocol family 17
[    1.420983] 9pnet: Installing 9P2000 support
[    1.421808] Key type dns_resolver registered
[    1.555067] Freeing unused kernel memory: 212K
[    1.555694] This architecture does not have kernel memory protection.
[    1.556464] Run /init as init process
           _  _
          | ||_|
          | | _ ____  _   _  _  _
          | || |  _ \| | | |\ \/ /
          | || | | | | |_| |/    \
          |_||_|_| |_|\____|\_/\_/

               Busybox Rootfs

Please press Enter to activate this console.
/ # cat /proc/interrupts
           CPU0
 10:         85  SiFive PLIC  10  ttyS0
IPI0:         0  Rescheduling interrupts
IPI1:         0  Function call interrupts
IPI2:         0  CPU stop interrupts
/ # cat /proc/cpuinfo
processor	: 0
hart		: 0
isa		: rv64imafdcuh
mmu		: sv48

/ # dmesg | grep kvm
[    0.856104] kvm [1]: hypervisor extension available
[    0.856811] kvm [1]: host has 14 VMID bits
/ #
/ # [   22.767774] random: fast init done

/ #
/ # ./apps/lkvm-static run -m 128 -c2 --console serial -p "console=ttyS0 earlyco
n=uart8250,mmio,0x3f8" -k ./apps/Image --debug
  # lkvm run -k ./apps/Image -m 128 -c 2 --name guest-52
  Info: (riscv/kvm.c) kvm__arch_load_kernel_image:117: Loaded kernel to 0x80200000 (9076412 bytes)
  Info: (riscv/kvm.c) kvm__arch_load_kernel_image:129: Placing fdt at 0x81000000 - 0x87ffffff
  # Warning: The maximum recommended amount of VCPUs is 1
  Info: (virtio/mmio.c) virtio_mmio_init:325: virtio-mmio.devices=0x200@0x10000000:5
  Info: (virtio/mmio.c) virtio_mmio_init:325: virtio-mmio.devices=0x200@0x10000200:6
  Info: (virtio/mmio.c) virtio_mmio_init:325: virtio-mmio.devices=0x200@0x10000400:7
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.3.0-rc6-00027-g9731700e1faf (anup@anup-ubuntu64) (gcc version 8.2.0 (Buildroot 2018.11-rc2-00003-ga0787e9)) #1 SMP Tue Sep 3 10:35:02 IST 2019
[    0.000000] earlycon: uart8250 at MMIO 0x00000000000003f8 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] initrd not found or empty - disabling initrd
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x0000000087ffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x0000000087ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x0000000087ffffff]
[    0.000000] software IO TLB: mapped [mem 0x83e3c000-0x87e3c000] (64MB)
[    0.000000] riscv: ISA extensions acdfimsu
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] percpu: Embedded 18 pages/cpu s34776 r8192 d30760 u73728
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 31815
[    0.000000] Kernel command line:  console=ttyS0 rw rootflags=trans=virtio,version=9p2000.L,cache=loose rootfstype=9p init=/virt/init  ip=dhcp console=ttyS0 earlycon=uart8250,mmio,0x3f8
[    0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 52092K/129024K available (6061K kernel code, 381K rwdata, 1886K rodata, 213K init, 305K bss, 76932K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] plic: mapped 1024 interrupts with 2 handlers for 4 contexts.
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
[    0.000200] sched_clock: 64 bits at 10MHz, resolution 100ns, wraps every 4398046511100ns
[    0.171749] Console: colour dummy device 80x25
[    0.249844] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000)
[    0.388978] pid_max: default: 32768 minimum: 301
[    0.461782] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.567119] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.967436] rcu: Hierarchical SRCU implementation.
[    1.140233] smp: Bringing up secondary CPUs ...
[    1.258984] smp: Brought up 1 node, 2 CPUs
[    1.398435] devtmpfs: initialized
[    1.448890] random: get_random_u32 called from bucket_table_alloc.isra.10+0x4e/0x160 with crng_init=0
[    1.552528] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.615323] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    1.673717] NET: Registered protocol family 16
[    2.189720] vgaarb: loaded
[    2.222607] SCSI subsystem initialized
[    2.284195] usbcore: registered new interface driver usbfs
[    2.341681] usbcore: registered new interface driver hub
[    2.403564] usbcore: registered new device driver usb
[    2.498567] clocksource: Switched to clocksource riscv_clocksource
[    2.939520] NET: Registered protocol family 2
[    2.979238] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    3.065841] TCP established hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    3.147607] TCP bind hash table entries: 1024 (order: 2, 16384 bytes, linear)
[    3.211371] hrtimer: interrupt took 13547200 ns
[    3.257799] TCP: Hash tables configured (established 1024 bind 1024)
[    3.329687] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    3.383520] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    3.460993] NET: Registered protocol family 1
[    3.595519] RPC: Registered named UNIX socket transport module.
[    3.644880] RPC: Registered udp transport module.
[    3.677386] RPC: Registered tcp transport module.
[    3.718955] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.762656] PCI: CLS 0 bytes, default 64
[    3.913379] kvm [1]: hypervisor extension not available
[    4.029738] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[    4.273208] NFS: Registering the id_resolver key type
[    4.295392] Key type id_resolver registered
[    4.330170] Key type id_legacy registered
[    4.360834] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    4.433719] 9p: Installing v9fs 9p2000 file system support
[    4.489825] NET: Registered protocol family 38
[    4.514998] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    4.575568] io scheduler mq-deadline registered
[    4.614369] io scheduler kyber registered
[    7.917465] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    7.983990] printk: console [ttyS0] disabled
[    8.069675] 3f8.U6_16550A: ttyS0 at MMIO 0x3f8 (irq = 3, base_baud = 115200) is a 16550A
[    8.240193] printk: console [ttyS0] enabled
[    8.240193] printk: console [ttyS0] enabled
[    8.351506] printk: bootconsole [uart8250] disabled
[    8.351506] printk: bootconsole [uart8250] disabled
[    8.471625] 2f8.U6_16550A: ttyS1 at MMIO 0x2f8 (irq = 4, base_baud = 115200) is a 16550A
[    8.603568] 3e8.U6_16550A: ttyS2 at MMIO 0x3e8 (irq = 6, base_baud = 115200) is a 16550A
[    8.726135] 2e8.U6_16550A: ttyS3 at MMIO 0x2e8 (irq = 7, base_baud = 115200) is a 16550A
[    8.839758] [drm] radeon kernel modesetting enabled.
[    9.137140] loop: module loaded
[    9.215478] libphy: Fixed MDIO Bus: probed
[    9.433530] e[   49.371354] random: crng init done
1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    9.532721] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    9.623184] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    9.705122] ehci-pci: EHCI PCI platform driver
[    9.755566] ehci-platform: EHCI generic platform driver
[    9.822157] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    9.938991] ohci-pci: OHCI PCI platform driver
[    9.995247] ohci-platform: OHCI generic platform driver
[   10.122915] usbcore: registered new interface driver uas
[   10.169577] usbcore: registered new interface driver usb-storage
[   10.249165] mousedev: PS/2 mouse device common for all mice
[   10.435596] usbcore: registered new interface driver usbhid
[   10.497111] usbhid: USB HID core driver
[   10.589728] NET: Registered protocol family 10
[   10.741662] Segment Routing with IPv6
[   10.765099] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[   10.861296] NET: Registered protocol family 17
[   10.917137] 9pnet: Installing 9P2000 support
[   11.098784] Key type dns_resolver registered
[   11.307381] Sending DHCP requests ., OK
[   11.456810] IP-Config: Got DHCP answer from 192.168.33.1, my address is 192.168.33.15
[   11.573144] IP-Config: Complete:
[   11.609328]      device=eth0, hwaddr=02:15:15:15:15:15, ipaddr=192.168.33.15, mask=255.255.255.0, gw=192.168.33.1
[   11.749868]      host=192.168.33.15, domain=, nis-domain=(none)
[   11.819542]      bootserver=192.168.33.1, rootserver=0.0.0.0, rootpath=
[   11.828083]      nameserver0=192.168.33.1
[   12.162799] VFS: Mounted root (9p filesystem) on device 0:13.
[   12.301665] devtmpfs: mounted
[   12.480813] Freeing unused kernel memory: 212K
[   12.528730] This architecture does not have kernel memory protection.
[   12.599440] Run /virt/init as init process
Mounting...
[   15.682709] random: fast init done
/ #
/ #
/ # cat /proc/interrupts
           CPU0       CPU1
  1:        172          0  SiFive PLIC   6  virtio1
  2:          9          0  SiFive PLIC   7  virtio2
  3:         68          0  SiFive PLIC   1  ttyS0
  5:        134          0  SiFive PLIC   5  virtio0
IPI0:       839        769  Rescheduling interrupts
IPI1:         0          1  Function call interrupts
IPI2:         0          0  CPU stop interrupts
/ #
/ #
/ # cat /proc/cpuinfo
processor	: 0
hart		: 0
isa		: rv64imafdcu
mmu		: sv48

processor	: 1
hart		: 1
isa		: rv64imafdcu
mmu		: sv48

/ #
/ #
/ # ping 192.168.33.1
PING 192.168.33.1 (192.168.33.1): 56 data bytes
64 bytes from 192.168.33.1: seq=0 ttl=64 time=73.298 ms
64 bytes from 192.168.33.1: seq=1 ttl=64 time=22.973 ms
64 bytes from 192.168.33.1: seq=2 ttl=64 time=6.421 ms
64 bytes from 192.168.33.1: seq=3 ttl=64 time=12.335 ms
64 bytes from 192.168.33.1: seq=4 ttl=64 time=5.666 ms
64 bytes from 192.168.33.1: seq=5 ttl=64 time=6.561 ms
64 bytes from 192.168.33.1: seq=6 ttl=64 time=8.076 ms
^C
--- 192.168.33.1 ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 5.666/19.332/73.298 ms
/ #
/ #
/ # poweroff -f
[   45.889505] reboot: Power down

  # KVM session ended normally.
/ #
/ #

Run Xvisor on qemu

This section of readme explains how to build & run Xvisor for RISC-V VirtIO board emulated by QEMU.

Please follow the steps below to build & run Linux kernel with Busybox RootFS on Virt64 (paravirtualized RISC-V 64bit) Guest with Xvisor running on QEMU model (v4.2 or higher):

Build Xvisor & DTBs

1
2
3
$ cd <xvisor_source_directory>
$ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- generic-64b-defconfig 
$ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu-

The above commands will create build/vmm.bin

Build Basic Firmware

1
$ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -C tests/riscv/virt64/basic

Build linux kernel

1
2
3
4
5
$ cd <linux_source_directory>
$ cp arch/riscv/configs/defconfig arch/riscv/configs/tmp-virt64_defconfig
$ <xvisor_source_directory>/tests/common/scripts/update-linux-defconfig.sh -p arch/riscv/configs/tmp-virt64_defconfig -f <xvisor_source_directory>/tests/riscv/virt64/linux/linux_extra.config
$ make O=<linux_build_directory> ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- tmp-virt64_defconfig
$ make O=<linux_build_directory> ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- Image dtbs

Build rootfs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cp <xvisor_source_directory>/tests/common/busybox/busybox-<busybox_version>_defconfig <busybox_source_directory>/.config
cd <busybox_source_directory>
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- oldconfig
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- install
mkdir -p ./_install/etc/init.d
mkdir -p ./_install/dev
mkdir -p ./_install/proc
mkdir -p ./_install/sys
ln -sf /sbin/init ./_install/init
cp -f <xvisor_source_directory>/tests/common/busybox/fstab ./_install/etc/fstab
cp -f <xvisor_source_directory>/tests/common/busybox/rcS ./_install/etc/init.d/rcS
cp -f <xvisor_source_directory>/tests/common/busybox/motd ./_install/etc/motd
cp -f <xvisor_source_directory>/tests/common/busybox/logo_linux_clut224.ppm ./_install/etc/logo_linux_clut224.ppm
cp -f <xvisor_source_directory>/tests/common/busybox/logo_linux_vga16.ppm ./_install/etc/logo_linux_vga16.ppm
cd ./_install; find ./ | cpio -o -H newc > ../rootfs.img; cd -

The above commands will create rootfs.img

Create disk image for Xvisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ cd <xvisor_source_directory>
$ mkdir -p ./build/disk/tmp
$ mkdir -p ./build/disk/system
$ cp -f ./docs/banner/roman.txt ./build/disk/system/banner.txt
$ cp -f ./docs/logo/xvisor_logo_name.ppm ./build/disk/system/logo.ppm
$ mkdir -p ./build/disk/images/riscv/virt64
$ dtc -q -I dts -O dtb -o ./build/disk/images/riscv/virt64-guest.dtb ./tests/riscv/virt64/virt64-guest.dts
$ cp -f ./build/tests/riscv/virt64/basic/firmware.bin ./build/disk/images/riscv/virt64/firmware.bin
$ cp -f ./tests/riscv/virt64/linux/nor_flash.list ./build/disk/images/riscv/virt64/nor_flash.list
$ cp -f ./tests/riscv/virt64/linux/cmdlist ./build/disk/images/riscv/virt64/cmdlist
$ cp -f ./tests/riscv/virt64/xscript/one_guest_virt64.xscript ./build/disk/boot.xscript
$ cp -f <linux_build_directory>/arch/riscv/boot/Image ./build/disk/images/riscv/virt64/Image
$ dtc -q -I dts -O dtb -o ./build/disk/images/riscv/virt64/virt64.dtb ./tests/riscv/virt64/linux/virt64.dts
$ cp -f <busybox_rootfs_directory>/rootfs.img ./build/disk/images/riscv/virt64/rootfs.img
$ sudo apt-get install genext2fs
$ genext2fs -B 1024 -b 32768 -d ./build/disk ./build/disk.img

Launch qemu

1
$ qemu-system-riscv64 -cpu rv64,x-h=true -M virt -m 512M -display none -serial stdio -bios <opensbi_build_directory>/platform/qemu/virt64/firmware/fw_jump.bin -kernel ./build/vmm.bin -initrd ./build/disk.img -append "vmm.console=uart@10000000 vmm.rtcdev=rtc@101000 vmm.bootcmd=\"vfs mount initrd /;vfs run /boot.xscript;vfs cat /system/banner.txt\""

Kick Guest0 to start Basic Firmware

1
XVisor# guest kick guest0

Bind to virtual UART

XVisor# vserial bind guest0/uart0

Copy linux from NOR flash to RAM and start linux booting from RAM

1
2
3
[guest0/uart0] basic# autoexec
  (Note: "autoexec" is a short-cut command)
  (Note: The <xvisor_source_directory>/tests/riscv/virt64/linux/cmdlist file which we have added to guest NOR flash contains set of commands for booting linux from NOR flash)

Wait for Linux prompt to come-up and then try out some commands

1
[guest0/uart0] / # ls

Enter character seqence ‘ESCAPE+x+q” return to Xvisor prompt]

1
[guest0/uart0] / #

Reference

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
https://github.com/kvm-riscv/howto/wiki/KVM-RISCV64-on-QEMU
https://github.com/riscv/riscv-gnu-toolchain/blob/master/README.md
https://cnrv.io/articles/spike-linux
https://github.com/xvisor/xvisor
http://xhypervisor.org/index.php?page=download
https://risc-v-getting-started-guide.readthedocs.io/en/latest/linux-qemu.html
https://wiki.qemu.org/Documentation/Platforms/RISCV
https://github.com/cambridgehackers/riscv-qemu
https://www.cnblogs.com/javaIOException/p/7525828.html
https://blog.csdn.net/shensen0304/article/details/95504258
https://www.riscv-mcu.com/article-article-show-id-404.html
https://github.com/riscv/opensbi/blob/master/docs/platform/qemu_virt.md

https://dzone.com/articles/multiple-binaries-with-gdbeclipse
https://mp.weixin.qq.com/s/ByXbn3gM7vai-mPdywn3Dg