Commit ccb900ec authored by Leonardo Solis's avatar Leonardo Solis
Browse files

added support to sauron

parent 9574082f
Version compiled with Intel `Altera Tools 16.0` **(Quartus upgraded & patched)** and targeting `harp2` machine. # Description
Version compiled with Intel `Altera OpenCL Compiler 16.0` (Quartus upgraded and patched) targeting the Arria10 reference board `a10gx`.
Source first: Source first:
```zsh ```zsh
source init_aoc_esa.sh % source init_170.sh
```
**ofdock_datapar_alt**: data-parallel version
**ofdock_taskpar_alt**: task-parallel version
# Update to a new branch
The FPGA program is made of 8 kernels:
``` ```
`GA` -> `IC` + `GG` + `LS` -> `Conform` -> `IE` + `IA` -> `Store`
```
The access to off-chip memory in order to update population and energy values is made in `GA`, `IC`, `GG`, `LS`, and `Store`.
The memory access seems to not be synchronized even though `mem_fence(CLK_GLOBAL_MEM_FENCE | CLK_CHANNEL_MEM_FENCE)` was used.
According to [this forum post](https://www.alteraforum.com/forum/showthread.php?t=56402), it is suggested to access off-chip memory using a single kernel.
A new branch called [`fusion`](https://gitlab/lvs/ofdock_altera/tree/fusion) is created where `GA`, `IC`, `GG`, `LS`, and `Store` are merged into a single kernel `GA`.
That way the design doesn't rely anymore on `mem_fence`s.
# Parallel versions
* [ofdock_datapar_alt](./ofdock_datapar_alt): data-parallel
* [ofdock_taskpar_alt](./ofdock_taskpar_alt): task-parallel
# This is created to automate the boring
# process of setting AOC in celebdil
# DO NOT FORGET to run it using:
# >source init_aoc_esa.sh
echo "======================================="
echo "Setting up Altera OpenCL SDK & Compiler"
echo "======================================="
echo ""
export LM_LICENSE_FILE=/opt/cad/keys/altera
echo "LM_LICENSE_FILE: "
echo $LM_LICENSE_FILE
echo " "
export PATH=/opt/altera-16.0_pro/quartus/bin/:$PATH
echo "PATH: "
echo $PATH
echo " "
export ALTERAOCLSDKROOT=/opt/altera-16.0_pro/hld
echo "ALTERAOCLSDKROOT: "
echo $ALTERAOCLSDKROOT
echo " "
export AOCL_BOARD_PACKAGE_ROOT=/opt/altera-16.0_pro/hld/board/Proc10A_16.0.2
echo "AOCL_BOARD_PACKAGE_ROOT: "
echo $AOCL_BOARD_PACKAGE_ROOT
echo " "
source $ALTERAOCLSDKROOT/init_opencl.sh
echo " "
echo "List of available boards "
aoc --list-boards
echo " "
echo "Altera OpenCL SDK version: "
aocl version
echo " "
echo "Altera OpenCL Compiler version: "
aoc --version
echo " "
...@@ -326,12 +326,6 @@ else ...@@ -326,12 +326,6 @@ else
endif endif
# ============================= # =============================
# This flag was adapted only for host-compilation on celebdil using HARP2 BSP.
# Kernel compilation (emu, hw) must use the default flags.
#AOCL_LINK_CONFIG_ADAPTED := -L/opt/cad/altera/altera-16.0/hld/linux64/lib -L/opt/cad/altera/HARP2/BSP/bdw_fpga_pilot_opencl_bsp_v1.0.2/host/linux64/lib -L$AALSDK/mybuild/myinstall/usr/local/lib -L/opt/cad/altera/altera-16.0/hld/host/linux64/lib -Wl,--no-as-needed -lalteracl
# Make it all! # Make it all!
all : $(TARGET_DIR)/$(TARGET) all : $(TARGET_DIR)/$(TARGET)
...@@ -344,7 +338,6 @@ all : $(TARGET_DIR)/$(TARGET) ...@@ -344,7 +338,6 @@ all : $(TARGET_DIR)/$(TARGET)
# $(foreach L,$(LIBS),-l$L) \ # $(foreach L,$(LIBS),-l$L) \
# -o $(TARGET_DIR)/$(TARGET) # -o $(TARGET_DIR)/$(TARGET)
# Enable this when compiling host in HARP2 machine
$(TARGET_DIR)/$(TARGET) : Makefile $(SRCS) $(INCS) $(TARGET_DIR) $(TARGET_DIR)/$(TARGET) : Makefile $(SRCS) $(INCS) $(TARGET_DIR)
$(ECHO)$(CXX) $(CPPFLAGS) $(CXXFLAGS) -fPIC $(foreach D,$(INC_DIRS),-I$D) \ $(ECHO)$(CXX) $(CPPFLAGS) $(CXXFLAGS) -fPIC $(foreach D,$(INC_DIRS),-I$D) \
$(AOCL_COMPILE_CONFIG) $(SRCS) $(AOCL_LINK_CONFIG) \ $(AOCL_COMPILE_CONFIG) $(SRCS) $(AOCL_LINK_CONFIG) \
...@@ -357,24 +350,6 @@ $(TARGET_DIR)/$(TARGET) : Makefile $(SRCS) $(INCS) $(TARGET_DIR) ...@@ -357,24 +350,6 @@ $(TARGET_DIR)/$(TARGET) : Makefile $(SRCS) $(INCS) $(TARGET_DIR)
@echo "================================" @echo "================================"
@echo " " @echo " "
## Adapted only for emulating HARP2 BSP on celebdil
## Disable this when compiling host in HARP2 machine
#$(TARGET_DIR)/$(TARGET) : Makefile $(SRCS) $(INCS) $(TARGET_DIR)
# $(ECHO)$(CXX) $(CPPFLAGS) $(CXXFLAGS) -fPIC $(foreach D,$(INC_DIRS),-I$D) \
# $(AOCL_COMPILE_CONFIG) $(SRCS) $(AOCL_LINK_CONFIG_ADAPTED) \
# $(foreach D,$(LIB_DIRS),-L$D) \
# $(foreach L,$(LIBS),-l$L) \
# -o $(TARGET_DIR)/$(TARGET) $(DOCK_DEBUG_FLAG) $(DEV) $(DOCK_PROG) $(ENABLE_KERNELS) $(REP)
# @echo " "
# @echo "================================"
# @echo "INFO: host binary is under: bin/"
# @echo "================================"
# @echo " "
# Docking parameters # Docking parameters
PDB := 3ptb PDB := 3ptb
NRUN := 10 NRUN := 10
...@@ -385,9 +360,7 @@ NEV := 100000 ...@@ -385,9 +360,7 @@ NEV := 100000
# Device main kernel source # Device main kernel source
DEV_DIRS := device DEV_DIRS := device
DEV_SRC := $(DEV_DIRS)/Krnl_GA.cl DEV_SRC := $(DEV_DIRS)/Krnl_GA.cl
#BOARD_HARP2 := bdw_fpga_v1.0 BOARD := Proc10A_X115
#BOARD_HARP2 := a10gx
BOARD_HARP2 := Proc10A_X115
## Make it all: compilation (host & device) + run emulation ## Make it all: compilation (host & device) + run emulation
## Notice, it switches automatically to /bin folder ## Notice, it switches automatically to /bin folder
...@@ -400,7 +373,7 @@ BOARD_HARP2 := Proc10A_X115 ...@@ -400,7 +373,7 @@ BOARD_HARP2 := Proc10A_X115
# docking.aoco # docking.aoco
# docking.aocx # docking.aocx
keremu: $(DEV_SRC) keremu: $(DEV_SRC)
aoc -march=emulator -v --board $(BOARD_HARP2) $(DEV_SRC) -o $(TARGET_DIR)/docking.aocx $(REP) aoc -march=emulator -v --board $(BOARD) $(DEV_SRC) -o $(TARGET_DIR)/docking.aocx $(REP)
@echo " " @echo " "
@echo "==========================================================" @echo "=========================================================="
@echo "INFO: emulation files are under: bin/docking(.aoco)(.aocx)" @echo "INFO: emulation files are under: bin/docking(.aoco)(.aocx)"
...@@ -410,7 +383,7 @@ keremu: $(DEV_SRC) ...@@ -410,7 +383,7 @@ keremu: $(DEV_SRC)
# Run emulation # Run emulation
# Compilation (host & device) if not done previously # Compilation (host & device) if not done previously
# The working directory is changed # The working directory is changed
# So .dlg file is written there # So .dlg file is written in $(TARGET_DIR)
emu: $(TARGET_DIR)/$(TARGET) keremu emu: $(TARGET_DIR)/$(TARGET) keremu
cd $(TARGET_DIR) && \ cd $(TARGET_DIR) && \
CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./$(TARGET) -ffile ../input/$(PDB)/derived/$(PDB)_protein.maps.fld -lfile ../input/$(PDB)/derived/$(PDB)_ligand.pdbqt -nev $(NEV) -nrun $(NRUN) -gfpop 1 CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./$(TARGET) -ffile ../input/$(PDB)/derived/$(PDB)_protein.maps.fld -lfile ../input/$(PDB)/derived/$(PDB)_ligand.pdbqt -nev $(NEV) -nrun $(NRUN) -gfpop 1
...@@ -428,7 +401,7 @@ emu: $(TARGET_DIR)/$(TARGET) keremu ...@@ -428,7 +401,7 @@ emu: $(TARGET_DIR)/$(TARGET) keremu
# docking.aoco # docking.aoco
# No actual hardware is built (NO .aocx) # No actual hardware is built (NO .aocx)
kerrpt: $(DEV_SRC) kerrpt: $(DEV_SRC)
aoc --board $(BOARD_HARP2) $(DEV_SRC) -o $(TARGET_DIR)/docking.aoco $(REP) -c --report -g aoc --board $(BOARD) $(DEV_SRC) -o $(TARGET_DIR)/docking.aoco $(REP) -c --report -g
@echo " " @echo " "
@echo "=============================================================" @echo "============================================================="
@echo "INFO: report files are under: bin/docking/reports/report.html" @echo "INFO: report files are under: bin/docking/reports/report.html"
...@@ -442,7 +415,7 @@ kerrpt: $(DEV_SRC) ...@@ -442,7 +415,7 @@ kerrpt: $(DEV_SRC)
# docking.aoco # docking.aoco
# docking.aocx # docking.aocx
hw: hw:
aoc --board $(BOARD_HARP2) $(DEV_SRC) -o $(TARGET_DIR_HW)/docking.aocx aoc --board $(BOARD) $(DEV_SRC) -o $(TARGET_DIR_HW)/docking.aocx
@echo " " @echo " "
@echo "===========================================================" @echo "==========================================================="
@echo "INFO: FPGA bitstream is under: bin_hw/docking(.aoco)(.aocx)" @echo "INFO: FPGA bitstream is under: bin_hw/docking(.aoco)(.aocx)"
...@@ -450,7 +423,7 @@ hw: ...@@ -450,7 +423,7 @@ hw:
@echo " " @echo " "
hw_higheffort: hw_higheffort:
aoc --high-effort --board $(BOARD_HARP2) $(DEV_SRC) -o $(TARGET_DIR_HW)/docking.aocx aoc --high-effort --board $(BOARD) $(DEV_SRC) -o $(TARGET_DIR_HW)/docking.aocx
@echo " " @echo " "
@echo "===========================================================" @echo "==========================================================="
@echo "INFO: FPGA bitstream is under: bin_hw/docking(.aoco)(.aocx)" @echo "INFO: FPGA bitstream is under: bin_hw/docking(.aoco)(.aocx)"
...@@ -464,7 +437,7 @@ hw_higheffort: ...@@ -464,7 +437,7 @@ hw_higheffort:
# docking.aoco # docking.aoco
# docking.aocx # docking.aocx
hw_profile: hw_profile:
aoc --profile --board $(BOARD_HARP2) $(DEV_SRC) -o $(TARGET_DIR_HW_PROF)/docking.aocx aoc --profile --board $(BOARD) $(DEV_SRC) -o $(TARGET_DIR_HW_PROF)/docking.aocx
@echo " " @echo " "
@echo "================================================================================" @echo "================================================================================"
@echo "INFO: instrumented FPGA bitstream is under: bin_hw_profile/docking(.aoco)(.aocx)" @echo "INFO: instrumented FPGA bitstream is under: bin_hw_profile/docking(.aoco)(.aocx)"
...@@ -472,38 +445,70 @@ hw_profile: ...@@ -472,38 +445,70 @@ hw_profile:
@echo " " @echo " "
HARP2_DIR := copyharp2_taskpar # Execute on sauron server
HARP2_DIR_DOCK := ofdock_taskpar_alt # Host-recompilation
# Copy of host to $(TARGET_DIR_HW)
# The working directory is changed
# So .dlg file is written in $(TARGET_DIR_HW)
exe: $(TARGET_DIR)/$(TARGET)
cp $(TARGET_DIR)/$(TARGET) $(TARGET_DIR_HW)/ && \
cd $(TARGET_DIR_HW) && \
./$(TARGET) -ffile ../input/$(PDB)/derived/$(PDB)_protein.maps.fld -lfile ../input/$(PDB)/derived/$(PDB)_ligand.pdbqt -nrun 10
@echo " "
@echo "==========================================="
@echo "INFO: log file is under: bin_hw/docking.dlg"
@echo "==========================================="
@echo " "
# Execute and profile on sauron server
# Host-recompilation
# Copy of host to $(TARGET_DIR_HW_PROF)
# The working directory is changed
# So .dlg file is written in $(TARGET_DIR_HW_PROF)
exe-prof: $(TARGET_DIR)/$(TARGET)
cp $(TARGET_DIR)/$(TARGET) $(TARGET_DIR_HW_PROF)/ && \
cd $(TARGET_DIR_HW_PROF) && \
./$(TARGET) -ffile ../input/$(PDB)/derived/$(PDB)_protein.maps.fld -lfile ../input/$(PDB)/derived/$(PDB)_ligand.pdbqt -nev $(NEV) -nrun 10
@echo " "
@echo "==================================================="
@echo "INFO: log file is under: bin_hw_profile/docking.dlg"
@echo "==================================================="
@echo " "
COPY_DIR := copy_taskpar
COPY_DIR_DOCK := ofdock_taskpar_alt
# Delete an existing folder if previously created # Delete an existing folder if previously created
# Create a folder $(HARP2_DIR) in the parent directory # Create a folder $(COPY_DIR) in the parent directory
# Copy binaries into $(HARP2_DIR) folder, avoiding the (innecesary) compilation intermediate files # Copy binaries into $(COPY_DIR) folder, avoiding the (innecesary) compilation intermediate files
# Copy source files into $(HARP2_DIR) as host is built in the HARP2 front-end # Copy source files into $(COPY_DIR) as host is built in the HARP2 front-end
# Create a zip file out of the $(HARP2_DIR) folder # Create a zip file out of the $(COPY_DIR) folder
copyharp2: clean_copyharp2 copy: clean-copy
mkdir ../$(HARP2_DIR) mkdir ../$(COPY_DIR)
mkdir ../$(HARP2_DIR)/$(HARP2_DIR_DOCK) mkdir ../$(COPY_DIR)/$(COPY_DIR_DOCK)
mkdir ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/$(TARGET_DIR_HW) mkdir ../$(COPY_DIR)/$(COPY_DIR_DOCK)/$(TARGET_DIR_HW)
mkdir ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/$(TARGET_DIR_HW_PROF) mkdir ../$(COPY_DIR)/$(COPY_DIR_DOCK)/$(TARGET_DIR_HW_PROF)
cp $(TARGET_DIR_HW)/docking.* ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/$(TARGET_DIR_HW)/ cp $(TARGET_DIR_HW)/docking.* ../$(COPY_DIR)/$(COPY_DIR_DOCK)/$(TARGET_DIR_HW)/
cp $(TARGET_DIR_HW_PROF)/docking.* ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/$(TARGET_DIR_HW_PROF)/ cp $(TARGET_DIR_HW_PROF)/docking.* ../$(COPY_DIR)/$(COPY_DIR_DOCK)/$(TARGET_DIR_HW_PROF)/
cp Makefile ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/ cp Makefile ../$(COPY_DIR)/$(COPY_DIR_DOCK)/
cp defines.h ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/ cp defines.h ../$(COPY_DIR)/$(COPY_DIR_DOCK)/
cp -r device/ ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/ cp -r device/ ../$(COPY_DIR)/$(COPY_DIR_DOCK)/
cp -r host/ ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/ cp -r host/ ../$(COPY_DIR)/$(COPY_DIR_DOCK)/
cp -r input/ ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/ cp -r input/ ../$(COPY_DIR)/$(COPY_DIR_DOCK)/
cp -r wrapcl/ ../$(HARP2_DIR)/$(HARP2_DIR_DOCK)/ cp -r wrapcl/ ../$(COPY_DIR)/$(COPY_DIR_DOCK)/
cp -r ../common ../$(HARP2_DIR)/ cp -r ../common ../$(COPY_DIR)/
cd ../; zip -r $(HARP2_DIR).zip $(HARP2_DIR) cp ../conf_aoc160_sauron.sh ../$(COPY_DIR)/
cd ../; zip -r $(COPY_DIR).zip $(COPY_DIR)
@echo "================================================================================" @echo "================================================================================"
@echo "The next step is to transfer the .zip file to the HARP2 fron-end: " @echo "The next step is to transfer the .zip file to sauron server: "
@echo "> cd .." @echo "> cd .."
@echo "> scp copyharp2_taskpar.zip h2solis@fe-1.harp.pc2.uni-paderborn.de:~/ocladock/" @echo "> scp copy_taskpar.zip sauron:~/"
@echo "================================================================================" @echo "================================================================================"
@echo " " @echo " "
clean_copyharp2: clean-copy:
rm -rf ../$(HARP2_DIR) rm -rf ../$(COPY_DIR)
$(TARGET_DIR) : $(TARGET_DIR) :
......
# Important # Important
Before building, Altera tools must be initialized. Before building, Altera tools must be initialized.
```zsh ```zsh
source init_aoc_esa.sh % source init_170.sh
``` ```
# Compile host # Compile host
Executable "host" in put under ./bin Executable "host" in put under `./bin`:
```zsh ```zsh
make clean % make clean
make % make
``` ```
# Compile kernel for emulation # Compile kernel for emulation
Kernels are implemented in different .cl files Kernels are implemented in different *.cl files:
But are included in calc_initpop.cl
```zsh ```zsh
aoc -march=emulator -v --board s5_ref ./device/calc_initpop.cl -o bin/docking.aocx % aoc -march=emulator -v --board s5_ref ./device/calc_initpop.cl -o bin/docking.aocx
``` ```
```zsh ```zsh
aoc -march=emulator -Werror -v --board a10gx device/Krnl_GA.cl -o bin/docking.aocx -DEMULATOR % aoc -march=emulator -Werror -v --board a10gx device/Krnl_GA.cl -o bin/docking.aocx -DEMULATOR
``` ```
# Relaxed fp operations # Relaxed floating-point operations
```zsh ```zsh
aoc -march=emulator --fp-relaxed --fpc -v --board a10gx device/Krnl_GA.cl -o bin/docking.aocx % aoc -march=emulator --fp-relaxed --fpc -v --board a10gx device/Krnl_GA.cl -o bin/docking.aocx
``` ```
# Run ofdock (under ./bin) # Emulation (under ./bin)
Emulation: this will execute 1 run This will execute 1 run docking run:
```zsh ```zsh
env CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./host -ffile ../input_data/1hvr_vegl.maps.fld -lfile ../input_data/1hvrl.pdbqt env CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./host -ffile ../input_data/1hvr_vegl.maps.fld -lfile ../input_data/1hvrl.pdbqt
``` ```
```zsh ```zsh
env CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./host -ffile ../input_data/3ptb/derived/3ptb_protein.maps.fld -lfile ../input_data/3ptb/deri$ env CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./host -ffile ../input_data/3ptb/derived/3ptb_protein.maps.fld -lfile ../input_data/3ptb/derived/3ptb_ligand.pdbqt
``` ```
# FPGA: this will execute 1 run # FPGA-execution:
This will execute 1 docking run:
```zsh ```zsh
./host -ffile ../input_data/1hvr_vegl.maps.fld -lfile ../input_data/1hvrl.pdbqt -nrun 1 ./host -ffile ../input_data/1hvr_vegl.maps.fld -lfile ../input_data/1hvrl.pdbqt -nrun 1
``` ```
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment