2.1.2
- Version:
- 2.0.0
- Date:
- 8/5/2014
"Introduction"
The XSPRESS3 system contains a Xilinx Virtex-5 FPGA with two embedded PowerPC processors. PPC1 manages the DMA engines. PPC1 runs the Xilinx micro kernel and communicates to the Intel 64 bit Linux server PC by 1 GBit Ethernet, TCP sockets. Bulk data and event lists to be histogrammed are sent from the firmware to the Server PC by 10G Ethernet, UDP.
Software Overview
The software API is via the libxspress3.a which provides C calling functions with names xsp3_*.
Setup procedure
The library supports running 1 or more XSPRESS3 systems. Each system can consists of 1 or more FEM cards. To connect to a system of 1 or more boards use the command xsp3_config This returns a handle which should be used to identify the system to all subsequent calls. Next setup the clock source. For normal operation the ADC and data processing clock is set to use the Crystal Oscillator module on the ADC board using xsp3_clocks_setup(path, -1, XSP3_CLK_SRC_XTAL, XSP3_CLK_FLAGS_MASTER | XSP3_CLK_FLAGS_NO_DITHER, 0) For digital only testing it is possible to generate the processing clock from the FEM card on board clocks. For multi board systems, the master board will be set to use the crystal and all the others will use a clock looped from this board. See xsp3_clocks_setup The processing register settings and BRAM contents are then loaded using xsp3_restore_settings The overall run flags should then be set to specify if whether playback data is to be used, whether scope mode debug data should be stored and whether scalers should be recorded. See xsp3_restore_settings The timing control register allows the acquisition enable/disable and time frame number to come from either software control (typically for a quick test) or from hardware (for accurately timed multi-frame experiments) See xsp3_set_glob_timeA and xsp3_set_glob_timeFixed
User Settings
The system now needs to need set up to match user specifiable values. In particular the energy windows and RoI positions need to be set to match the energy of the element(s) of interest. Two in windows scalers are provided per channel, typically used to measure the counts in the main characteristic peak of 1 or 2 elements See xsp3_set_window The Histogram data is usually set to collect the full spectrum in 4096 energy bins. Various debug modes are available, but not required for user data. To reduce the data volume, it is also possible to collect regions of interest around the peaks of interest. In this case the total size of each spectrum is reduced from 12 bits = 4096 bins to 11..1 bits => 2048 .. 2 bins. First the Spectra size is set using xsp3_format_run and then if required RoI are set using xsp3_set_roi
Collection
Once setup the histogramming memory is cleared using xsp3_histogram_clear. Then the system is started using xsp3_histogram_start If the timing control has been set to a fixed frame from software control, counting then starts. If the timing control has been set to use the external timing inputs, counting is armed, but will occur only when the timing veto input is asserted. If the acqusition is to be software timed, wait for the desired time and then counting disabled using xsp3_histogram_stop If timing if being controlled by hardware, the external timing generator must determine when the exposure is finished. After this call xsp3_histogram_stop to flush remaining data.
Data Access
The histogram data is stored in /dev/shm shared memory on the Linux server. It can be read at any time, accepting that data read will be slightly stale while histogramming is running. At the end of the experiment, after calling xsp3_histogram_stop, poll xsp3_histogram_is_busy twice to check that the histogramming threads have flushed any network stack buffers. The histogram data can then read read using any of xsp3_histogram_read4d, xsp3_histogram_read3d or xsp3_histogram_read_chan The scaler data is stored in a hardware counter during an acquistion and then transfered to DRAM on the FEM at the end of each frame. For multi-frame experiments, data can be read from the previous frame while the next is counting. To check the DMA channel has written data to memory use xsp3_dma_check_desc The raw scaler data is then read using xsp3_scaler_read
Register Layout
XSPRESS3 register space memory is mapped at 0xA0000000 on a 16 MByte boundary. It is split into 16 “channels” using the top 4 bits of my address space. Channel 0..7(8) will be physical channels and channel 15 is the global register. The remaining channels 10..14 are reserved for future use.
Within a channel, the top next 4 bits down select a region. Regions 0..8 (plus expansion) address block RAMs with varying sizes. Region 15 contains 30 registers. xspress3.h XSP3_REGIONS Address Bits
* 23..20 Channel Address * 19..16 Region (BRAM sel or Regs) * 15..2 BRAM addresses * 7 ..2 Register select *
Network Configuration
The default network setup is (excluding the site network connection):
* 1GBit Copper network for conbtrol communinication between the PC and the XSPRESS3 box. With more thean 1 XSPRESS3 box connected this netwrok uses a ethernt switch * A private network with 64 addresses allocated: * eth1 Link encap:Ethernet HWaddr d4:ae:52:7d:5f:84 * inet addr:192.168.0.1 Bcast:192.168.0.63 Mask:255.255.255.192 * inet6 addr: fe80::d6ae:52ff:fe7d:5f84/64 Scope:Link * UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 * RX packets:1567 errors:0 dropped:5766 overruns:0 frame:0 * TX packets:158 errors:0 dropped:0 overruns:0 carrier:0 * collisions:0 txqueuelen:1000 * RX bytes:173937 (169.8 KiB) TX bytes:37252 (36.3 KiB) * Interrupt:48 Memory:da000000-da012800 * * A 10GBit Fibre network for data transfer, point to point with 4 addresses allocated. With more that 1 XSPRESS3 box there would be multiple 10G Ports on the PC with multiple 4 address range subnets * * ifconfig eth2 * eth2 Link encap:Ethernet HWaddr 00:07:43:05:7c:65 * inet addr:192.168.0.65 Bcast:192.168.0.67 Mask:255.255.255.252 * inet6 addr: fe80::207:43ff:fe05:7c65/64 Scope:Link * UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 * RX packets:0 errors:0 dropped:0 overruns:0 frame:0 * TX packets:702 errors:0 dropped:0 overruns:0 carrier:0 * collisions:0 txqueuelen:1000 * RX bytes:0 (0.0 B) TX bytes:154963 (151.3 KiB) * Interrupt:41 Memory:dd7fe000-dd7fefff * * Note the carefully picked subnet masks etc and the MTU 9000 * We then have a script that should be executed automatically at boot. * * cat /etc/init.d/xspress3.sh #!/bin/bash # # static-arp This is to register a static ARP address in the arp table at boot # # Kept as simple as possible hopefully this will auto register the associated # MAC with the private network address to allow the machine to communicate with the # test boards for xspress3 # # Derived from work by Duncan Russell, by William Helsby PATH=/sbin:/bin:/usr/bin:/usr/sbin arp -i eth2 -s 192.168.0.66 02:00:00:00:00:00 #route -v add -host 192.168.0.66 eth2 # Setting default and max buffer sizes for networking. sysctl -w net.core.rmem_max=1073741824 sysctl -w net.core.rmem_default=1073741824 * *
Error Codes
The following error codes maybe returned; xspress3.h XSP3_ERROR_CODES
"Revison History"
1.0.0 Original Release. 2.0.0 8/5/2014 API supporting event assembly processing in software and scalers in software. Note: User code should now check for histogramming threads busy using xsp3_histogram_is_any_busy rather than xsp3_histogram_is_busy as new function check all cards and all channels as necessary. User code should use xsp3_scaler_check_progress to check progress through time frames.
