OpenSPARC T1: Architectural Transplant Instructions

Introduction

This document describes the installation and usage of Architectural Transplant package for the OpenSPARC T1. The contents assume familiarity with compiling and running OpenSPARC RTL models and using Simics/Niagara.

The basic components are a computer-generated assembly file that contains the checkpoint to be loaded (and a utility to generate this assembly file), a TCP socket connection to the full-system simulator (written in C, through the Verilog PLI), and a TCP socket listener module on the full-system simulator.

Requirements

The Transplant package requires the following platform software:

• Sun OpenSPARC T1 RTL package version 1.4
• Simics/Niagara (available from http://www.simics.net)
• A Verilog simulator with VPI (3rd generation PLI) support

We have developed and tested Transplants on the following system configuration:

• SuSE Linux 10.1 on i686 hardware
• Simics version 3.0.27 running a single-strand Niagara system model.
• GCC 3.4.4
• Perl 5.8.8
• Synposys VCS release Y-2006.06

We have not yet tested compilation on Solaris or with other Verilog simulators (i.e., Cadence Verilog or ModelSim). Also, we have not tried transplants with the latest (v1.5 as of this writing) OpenSPARC T1 release.

Overview of changes

The Transplant package contains the following changes to the OpenSPARC T1 RTL release:

• New Verilog monitor module for reading simulation command-line arguments and initiating transplants (verif/env/cmu/transplant.v).
• New Verilog PLI library (in tools/pli/transplant/) for socket communications to the full-system simulator and slamming state which cannot be easily initialized by assembly code. Changes to makefiles to compile the new library.
• Utility for generating custom verification tests from architectural state checkpoints (tools/pli/transplant/gen_checkpoint).
• Assembly routines for loading architectural register state into an OpenSPARC processor and starting execution at a checkpoint (verif/diag/assembly/transplant/transplant_routines.s).
• Modified DIMM model that can be switched to supply data from an external full-system simulator's memory image (verif/model/verilog/mem/dram/dimm.v).
• Simics/Niagara module for socket communications to generate checkpoints and provide memory (DRAM) state to the transplanted RTL model (simics-transplant/).
• Modifications to the LSU to support Simics's "alternate" address for the console serial port (PA 0x1f10000000 instead of PA 0x9810000000). This redirects requests to that address range to the IOB for processing.

Installation and compilation

Package Installation

1. Download, install, and test a copy of the OpenSPARC T1 RTL v1.4 using the core1 (single-strand, single-core) configuration with sims. Make sure the OpenSPARC T1 model compiles and can run assembly test programs such as those in core1_mini.
2. Download the Transplant release tarball.
3. Enter your $DV_ROOT directory and untar the tarball (note: some files from the original OpenSPARC T1 RTL distribution will be overwritten and some new files and directories will be added).

PLI Module Compilation

Recompile the PLI modules for your model with the following commands (note: command is shown for Synopsys VCS. Substitute ncv for vcs to compile with Cadence Verilog):

1. mkplilib clean
2. mkplilib vcs
3. Recompile the OpenSPARC RTL core1 model using sims. The updated model should link with the new transplant PLI library and run the verification programs from before without change. We used the following command:

   sims -sim_type=vcs -novera_build -novera_run -tg_seed 1 \
                -sys=core1 -nosas -group=core1_transplant

The Transplant module and other default PLI modules should build without warnings or errors. Furthermore, the gen_checkpoint program, which generates assembly-language checkpoint files is compiled and left in tools/pli/transplant/

Simics Module Compilation

1. Install and boot a copy of Simics/Niagara using the instructions that come with Simics. Running Simics and making Simics checkpoints is beyond the scope of this document.
2. Enter the $DV_ROOT directory and make a symbolic link to your Simics "workspace" directory that was part of the Simics installation named $DV_ROOT/workspace.
3. Enter the $DV_ROOT/simics-transplant/ directory.
4. Edit makefile.defs and modify all variables to match your environment. In particular, SIMICS_PATH should point to your Simics installation, GCC_PATH should point to your gcc directory, and GCC and GXX should point to your systems gcc and g++ binaries, respectively.
5. Compile the Simics transplant module by running make. If successful, the final line should show that the file transplant_mod.so was copied to $SIMICS_PATH/x86-linux/lib.

Preparing a checkpoint

The checkpoint consists of a running full-system simulator (for the memory image) and an assembly program containing checkpointed architectural register state. In order to generate a checkpoint, do the following steps:

1. Enter the $DV_ROOT/simics-transplant/ directory
2. Edit launch.simics and change the read-configuration command to point to the Simics/Niagara full-system checkpoint that you want to run on the RTL model.
3. Run ./run_transplant.sh, which will start Simics, load the transplant module, and start a TCP listening socket on port 8000
4. You can test the TCP socket by telnetting to the Simics instance and typing PC to print the PC and NPC register values from Simics. Example output below is in blue, although the PC/NPC values in your checkpoint will probably be different:

             $ telnet localhost 8000
             Trying 127.0.0.1...
             Connected to localhost.
             Escape character is '^]'.
             PC
             PC 197a8 197ac

   Press enter again to close the connection. Take a note of the starting PC for later usage (in this case 197a8).
5. Generate a checkpoint assembly file with the gen_checkpoint program, which takes a hostname:portnumber and output file as arguments. Example below:

   $DV_ROOT/tools/pli/transplant/gen_checkpoint  localhost:8000 \
                  $DV_ROOT/verif/diag/assembly/transplant/my_checkpoint.s

   On a successful run, there should be no output to the screen, but the output assembly file should be created. Note that the generated checkpoint file #include's transplant_routines.s, so the checkpoint file must be generated in the same directory as that assembly file.

Running a transplant

Follow the steps in Preparing a Checkpoint (above). If the checkpoint has not changed, you can reuse the generated assembly file multiple times. However, the full-system simulator must be running for the transplant to work. Also, have a compiled version of the RTL model with transplant support working, from the steps in PLI module compilation (above).

Run a copy of the RTL model with sims, using the additional arguments as follows:

-sim_run_args=+slam_host=localhost:8000
-sim_run_args=+slam_trigger_pc=197a8
-sim_run_args=+ictag_check_off=1
my_checkpoint.s

Note that the ictag_check_off argument is specified because transplants invalidate the caches, which confuses the I-cache tag check code because it keeps its own internal records of the I-cache tags.

For example, in our environment at CMU, we use the following command:

      sims \
       -vcs_run \
       -sys=core1 \
       -vcs_rel_name=core1_2007_08_14_0 \
       -regress_id=2007_08_14_0 \
       -alias=transplant_1:model_core1:core1_transplant:0 \
       -dv_root=/afs/scotch/usr/jsmolens/research/opensparc/transplant/os14 \
       -model_dir=/afs/scotch/usr/jsmolens/research/opensparc/transplant/os14 \
       -result_dir=/afs/scotch.ece.cmu.edu/project/first/transplant/os14 \
       -sims_config=/afs/scotch/usr/jsmolens/research/opensparc/transplant/os14/tools/src/sims/sims.config \
       -group_name=core1_transplant \
       -regress_date=2007_08_14 \
       -regress_time=11_06_22 \
       -sim_run_args=+slam_host=172.19.128.15:8000 \
       -sim_run_args=+slam_trigger_pc=197a8 \
       -sim_run_args=+ictag_check_off=1 \
       transplant_1.s \
       -novera_run \
       -tg_seed 1 \
       -sim_run_args=+spc_pipe=0 \
       -sys=core1 \
       -nosas \
       -vcs_build_args=-j6 \
       -nobuild \
       -regress \
       -nouse_oolm
      

Customize the arguments in italics to match your environment. In particular, the hostname that runs the full-system simulator (this may be a remote machine) and the starting PC, recorded earlier, in hexadecimal, but without a leading 0x must both be set. Finally, set the assembly regression file to the file created by gen_checkpoint.

The simulator should run for roughly 20K cycles, loading checkpoint state, before dropping into the transplanted program. At this point, a TCP connection will be established with the remote full-system simulator and the transplanted system will begin fetching instructions and data from the remote machine. Memory read/write activity should be visible from the Simics command window.

Execution will finish when the RTL model hits the maximum number of cycles (unlike verification assembly programs, there is no good/bad trap called within Solaris). You can run for a longer amount of time by setting -max_cycles=N on the sims command line. If you run long enough, you may observe output on the Simics console (if your program writes to the console). You can also monitor execution in the sim.log output file when the simulation terminates.

Contact Info

The transplant code is written by Jared Smolens with help from Eric Chung, both graduate students from the Computer Architecture Lab at Carnegie Mellon. Both students are advised by James C. Hoe. This project is an extension of the ProtoFlex project. Please post questions or comments regarding OpenSPARC transplants on the OpenSPARC general discussions forum.