fix(IPrefetchPipe): consider backend exception as part of itlb exception (#4423)

`s1_exception_out` is for prefetch s2 only, but we want backend
exception to be considered as part of itlb exception

fix(IPrefetchPipe): consider backend exception as part of itlb exception (#4423)

`s1_exception_out` is for prefetch s2 only, but we want backend
exception to be considered as part of itlb exception and sent to
waylookup, so we merge it to `s1_itlb_exception`

show more ...

feat(ICache): ECC error injection (#4044)

This PR is part of *RAS(Reliability, Accessibility, Serviceability)* error recovery features.

- Add a series of mmio-mapped CSR to control ICache ECC check

feat(ICache): ECC error injection (#4044)

This PR is part of *RAS(Reliability, Accessibility, Serviceability)* error recovery features.

- Add a series of mmio-mapped CSR to control ICache ECC check & ECC inject features
- Implement ICache ECC injection
- M-state software can write `eccctrl` to trigger error injection to meta/dataArray, next read can trigger auto-recovery (implemented in #3899)
- Remove custom CSR `Sfetchctl`

# Details
## CSR
The base address of the added mmio-mapped CSR is `0x38022080` and the registers is defined as below:
```
64 10 7 4 2 1 0
0x00 eccctrl | WARL | ierror | istatus | itarget | inject | enable |

64 PAddrBits-1 0
0x08 ecciaddr | WARL | paddr |
```
| CSR | field | desp |
| --- | --- | --- |
| eccctrl | enable | ECC check enable |
| eccctrl | inject | ECC inject enable (write 1 to trigger injection, read always 0) |
| eccctrl | itarget | ECC inject target<br>0: metaArray<br>1: rsvd<br>2: dataArray<br>3: rsvd |
| eccctrl | istatus | ECC inject status (read-only)<br>0: idle: inject controller idle, goes to working when received a inject request (i.e. write 1 to eccctrl.inject)<br>1: working: inject controller working, goes to injected when finished / error when failed<br>2: injected, goes to idle after read<br>3: rsvd<br>4: rsvd<br>5: rsvd<br>6: rsvd<br>7: error: inject failed (check eccctl.ierror for reason), goes to idle after read |
| eccctrl | ierror | ECC error reason (read-only, valid only if `eccctrl.istatus==error`)<br>0: ECC check is not enabled (i.e. `!eccctrl.enable`)<br>1: inject target invalid (i.e. `eccctrl.itarget==rsvd`)<br>2: inject addr (i.e. `ecciaddr.paddr`) not in ICache<br>3: rsvd<br>4: rsvd<br>5: rsvd<br>6: rsvd<br>7: rsvd |
| ecciaddr | paddr | Physical address of the inject target |

## Inject method
```asm
$INJECT_ADDR:
# maybe do something else
ret

test:
la t0, $BASE_ADDR # load icache control base addr
la t1, $INJECT_ADDR # load inject addr
jalr ra, 0(t1) # jump to injected addr to load it i
sd t1, 8(t0) # set inject addr
la t2, (target << 2 | 1 << 1 | 1 << 0) # load inject target & inject enable & ecc enable
sd t1, 0(t0) # set inject enable & ecc enable
loop:
ld t1, 0(t0) # get ecc control state
andi t1, t1, (0b11 << (4+1)) # get high bits of inject state
beqz t1, loop # if is idle, or working, loop

addi t1, t1, -1 # t1 = inject_state[2:1] - 1
bnez t1, error # if is not injected, error or rsvd

jalr ra, 0(t1) # jump to injected addr to trigger error
j finish

error:
# handle error
finish:
# finish
```
Or, checkout https://github.com/OpenXiangShan/nexus-am/pull/48

show more ...

fix(ICache,ITLB): also flush itlb pipe when prefetchPipe s1_flush (#3996)

ITLB does not store `gpaddr` to save resources, instead it takes
`gpaddr` from L2TLB when gpf occurs, which poses a two-opt

fix(ICache,ITLB): also flush itlb pipe when prefetchPipe s1_flush (#3996)

ITLB does not store `gpaddr` to save resources, instead it takes
`gpaddr` from L2TLB when gpf occurs, which poses a two-option
requirement for the requestor (i.e. IPrefetchPipe):
1. resend the same `itlb.io.req.vaddr` until `itlb.io.resp.miss` is
pulled down
2. flush gpf entry in ITLB by pulling up `itlb.io.flushPipe`

Otherwise, ITLB is unable to handle the next gpf and the core hangs.

However, the first point cannot be guaranteed during the speculative
execution, as IPrefetchPipe sends request to ITLB at s0 stage and may
receive a flush request from BPU s3 stage, IFU or Backend at s1 stage,
then the same vaddr is never resend to ITLB.

Therefore, we must ensure that ITLB is flushed synchronously when
IPrefetchPipe s1 stage is flushed, thus satisfying the second point.
This PR implements this.

show more ...

refactor(ICache): refactor code style & eliminate IDE warnings (#3947)

- add type annotation to public members
- add private qualifier to other members
- add `asUInt` to shift results (Bits)
- fi

refactor(ICache): refactor code style & eliminate IDE warnings (#3947)

- add type annotation to public members
- add private qualifier to other members
- add `asUInt` to shift results (Bits)
- fix typo
- remove unused imports
- rename `ICacheMSHR.waymask` to `way` since it is not a mask
- use `idxBits` for `log2Up(nSets)`
- use `wayBits` for `log2Up(nWays)`
- use `foreach` instead of `map` when return value is not needed
- use `{}` instead of `()` for multi-line `foreach` and `map`

The generated verilog is checked and is identical with the original
(except `waymask` -> `way` & order changes).

show more ...

fix(CSR): correct the width of PC pgaddr for inst fetch exception (#3795)

We found that the CSR mtval2 truncates the high bits of gpaddr when GPF
occurs in instruction fetching. Actually, there is

fix(CSR): correct the width of PC pgaddr for inst fetch exception (#3795)

We found that the CSR mtval2 truncates the high bits of gpaddr when GPF
occurs in instruction fetching. Actually, there is an GPAMem which
storages the whole 64-bit gpaddr, but it does not pass to CSR correctly,
due to incorrect width of trapPCGPA in module NewCSR and bundle
TrapEntryEventInput. This patch fixes this.

---------

Co-authored-by: ngc7331 <[email protected]>

show more ...

refactor(Frontend): add ExceptionType.hasExcaption wrapper (#3866)

https://github.com/OpenXiangShan/XiangShan/pull/3787#discussion_r1818322915

fix(ICache): cancel prefetch when there is exception from backend (#3787)

style(Frontend): use scalafmt formatting frontend (#3370)

Format frontend according to the scalafmt file drafted in #3061.

timing(IPrefetch): add 1 cycle to s2_finish (#3545)

Cut critical path prefetchPipe s2 -> toMSHRArbiter.valid(i) ->
toMSHR.paddr -> missUnit hit -> missUnit.req.ready -> prefetchPipe
toMSHRArbiter.

timing(IPrefetch): add 1 cycle to s2_finish (#3545)

Cut critical path prefetchPipe s2 -> toMSHRArbiter.valid(i) ->
toMSHR.paddr -> missUnit hit -> missUnit.req.ready -> prefetchPipe
toMSHRArbiter.ready ***-> s2_finish ->*** s2_ready -> s1_ready ->
toFtq.ready
for timing.

This can be thought of as adding 1 cycle to the prefetchPipe s2_finish,
but only a minor performance change is expected, since the timing of
issuing the first miss request is unchanged, and the additional waiting
delay for subsequent miss requests can be hidden by the l2 cache access
delay.

show more ...

feat(trap): support m/htinst for specific G-stage translation (#3604)

According to RISC-V priv spec, mtinst/htinst could be always written
zero on trap into M/HS-mode, except for Guest-Page-Fault t

feat(trap): support m/htinst for specific G-stage translation (#3604)

According to RISC-V priv spec, mtinst/htinst could be always written
zero on trap into M/HS-mode, except for Guest-Page-Fault traps that meet
both of the following conditions:
- the trap is caused by a G-stage translation which supports VS-stage
translation
- a nonzero value is written to mtval2/htval

"isForVSnonLeafPTE" is used only in exceptional circumstances that gpf
happens in the G-stage translation which supports VS-stage translation,
such as searching the non-leaf pte of VS-stage.

This patch adds support for writing proper value to mtinst/htinst when
specific trap occurs. And bump the nemu.

show more ...

fix(ICache): MSHR also update meta_codes when updating waymasks (#3492)

ICache: fix metaArray ECC check (#3419)

Currently, metaArray ECC check is valid 2 cycles after request:

https://github.com/OpenXiangShan/XiangShan/blob/49162c9ab67070931573c1d4a372e2c858a72716/sr

ICache: fix metaArray ECC check (#3419)

Currently, metaArray ECC check is valid 2 cycles after request:

https://github.com/OpenXiangShan/XiangShan/blob/49162c9ab67070931573c1d4a372e2c858a72716/src/main/scala/xiangshan/frontend/icache/ICache.scala#L262

However, prefetchPipe s1 handshakes with both WayLookup and prefetchPipe
s2 assuming that all signals of the metaArray.io.readResp are valid 1
cycle after the request, resulting in the error.

Simply removing this RegEnable may lead to problems with long timing
paths (metaArray (sram) -> ECC check (xor reduction) -> prefetchPipe s1
(wire) -> wayLookup (bypass, wire) -> mainPipe s0 (wire) -> mainPipe s1
(reg)), so no.

This PR may result in case-specific errors not being checked out, which
in turn results in additional fetch requests being sent to the L2 cache,
but does not causes corrupted data being sent to the backend. See
discussion in notes:

https://github.com/OpenXiangShan/XiangShan/blob/8b87b8dcbfd5945c5bd7815eb5e569fec252ddc6/src/main/scala/xiangshan/frontend/icache/IPrefetch.scala#L279-L293

There are 2 more potential solutions described in an internal yuque
document, however, due to the complexity of implementation, area
overhead and other considerations, the current solution is considered to
be optimal.

show more ...

IPrefetch: fix s1 fsm for softPrefetch (#3433)

Do not bypass `m_enqWay` state to make sure that s1_waymasks (and maybe
other registers) is updated, whether is softPrefetch or not.

svpbmt: add simplified support (#3404)

Only the `pbmt` attribute is added and treated as MMIO when `pbmt` is NC
or IO.

---------

Co-authored-by: ngc7331 <[email protected]>

Frontend: implement prefetch.i support (RVA23 Zicbop) (#3396)

IPrefetch: disable IPrefetchPipe s2 stage if CSR does not enable iprefetch (#3372)

ICache: raise af if meta/data array ECC fail

In current design, meta/data array corruption does not raise any
exception (whether or not `io.csr_parity_enable === true.B`), which may
pose two probl

ICache: raise af if meta/data array ECC fail

In current design, meta/data array corruption does not raise any
exception (whether or not `io.csr_parity_enable === true.B`), which may
pose two problems:
1. When meta corrupt, `ptag` comparison result may be invalid, and thus
cache hit may be treated as a cache miss, thereby sending (pre)fetch
request to L2 cache incorrectly;
2. When meta/data/l2 corrupt, instruction data sent to the backend may
be invalid. Although the errors are sent to beu, which sends an
interrupt via plic, the timing of the interrupt is not as controllable
as an exception. It is therefore reasonable to mark invalid data as
access fault to keep it from execution.

This PR:
1. Raise af if meta/data array ECC fail (when `io.csr_parity_enable ===
true.B`), the priority of this af is lower than iTLB & PMP exceptions
2. Cancle (pre)fetching if meta array ECC fail (by merging
`meta_corrupt` exceptions to `s2_exception`)

Note:
RISC-V Machine ISA v1.13 (draft) introduced a "hardware error"
exception, described as:
> A Hardware Error exception is a synchronous exception triggered when
corrupted or uncorrectable data is accessed explicitly or implicitly by
an instruction. In this context, "data" encompasses all types of
information used within a RISC-V hart. Upon a hardware error exception,
the xepc register is set to the address of the instruction that
attempted to access corrupted data, while the xtval register is set
either to 0 or to the virtual address of an instruction fetch, load, or
store that attempted to access corrupted data. The priority of Hardware
Error exception is implementation-defined, but any given occurrence is
generally expected to be recognized at the point in the overall priority
order at which the hardware error is discovered.

Maybe it's better to raise hardware error instead of access fault when
ECC check failed. But it's draft and XiangShan backend does not
implement this exception code yet, so we still raise af here. This may
need to be modified in the future.

show more ...

Frontend: refactor exceptions to labels (#3354)

Combine `excp_pf`/`_gpf`/`_af` into `exception` to:
1. Reduce code redundancy and improve readability and maintainability
e.g. `!itlb_excp_af && !it

Frontend: refactor exceptions to labels (#3354)

Combine `excp_pf`/`_gpf`/`_af` into `exception` to:
1. Reduce code redundancy and improve readability and maintainability
e.g. `!itlb_excp_af && !itlb_excp_pf && !itlb_excp_gpf && !pmp_excp_af
&& !pmp_excp_mmio`
-> `exception === ExcedptionType.none && !mmio`
2. Select exceptions as they are generated (e.g. from iTLB/PMP, or
data/meta array ECC check) on a priority basis (e.g. iTLB over PMP),
ensuring that there is at most one exception in the pipeline (and on the
ports of iCache -> IFU)
3. Save a little bit of pipeline/WayLookup registers (i.e. 3 bit
`excp_pf`/`_gpf`/`_af` -> 2bit `exception`)

show more ...

ICache: cancel (pre)fetch request if port1 is mmio (#3319)

IPrefetch: fix s2_miss(1) typo (#3239)

Explained: (**NOT** has exception **AND NOT** hit) === miss === needs fetch

Performance change is expected, as prefetch is actually doing more work.

IPrefetch: do not pass exception(1) to WayLookup if !s1_doubleline (#3215)

Fixes bug mentioned:
https://github.com/OpenXiangShan/XiangShan/pull/3139#discussion_r1679178024

Analysis:
1. (expecte

IPrefetch: do not pass exception(1) to WayLookup if !s1_doubleline (#3215)

Fixes bug mentioned:
https://github.com/OpenXiangShan/XiangShan/pull/3139#discussion_r1679178024

Analysis:
1. (expected) In a doubleline request, port0 AND port1 finds guest page
fault(`io_itlb_x_resp_bits_excp_0_gpf_instr`), it is stored in
`itlbExcpGPF` register, enters WayLookup and is bypassed to
ICacheMainPipe (WayLookup is `empty` and `io_write` fires with `io_read`
fire). Finally it goes to backend
2. (expected) Backend send a redirect request and flushes
IPrefetch/WayLookup/ICacheMainPipe
3. (WRONG) After flush, this is a singleline request, so port1 does not
send request to itlb(`io_itlb_1_req_valid`) and thus not updated,
`io_itlb_x_resp_bits_excp_0_gpf_instr` remains `true.B`
4. (WRONG) This false-positive gpf enters WayLookup and is bypassed to
ICacheMainPipe
5. (expected) However, ICacheMainPipe finds `s2_doubleline` is
`fasle.B`, so it drops results from port1, so no gpf goes to backend.
6. (expected) After so many requests, circular pointer in WayLookup
overflows and returns to the location where the gpf was written to, so
it reads gpd again
7. (expected) This time, `s2_doubleline` is `true.B`, so gpf goes to
backend and finally causes error.

Solution:
1. Flush tlb results when `io_flush === true.B`. This might require
modifications to both the IPrefetch and TLB, we may address it later.
2. **Drop port1 results before it enqueues into WayLookup, instead of
when it is sent to IFU (after dequeues from WayLookup)**

show more ...

Frontend: cut waylookup gpaddr (#3139)

Currently, `gpaddr` is used only when guest page fault occurs, so it
should be possible to not store every `gpaddr` in WayLookup (as well as
in GPAMem).

C

Frontend: cut waylookup gpaddr (#3139)

Currently, `gpaddr` is used only when guest page fault occurs, so it
should be possible to not store every `gpaddr` in WayLookup (as well as
in GPAMem).

Considering that every guest page fault should issue a redirect, and
thus flush WayLookup/MainPipe/IPrefetch, we should be able to store only
the first guest page fault and its `gpaddr` in WayLookup.

By doing this, we can save `2×nWayLookupSize - 1 = 63` registers that
are used to store `gpaddr`.

This PR also includes some style changes for WayLookup to satisfy
linter.

show more ...

IPrefetch: MSHR should update IPrefetch s1 waymask (#3122)

Fixes MC-Linux CI fail:
https://github.com/OpenXiangShan/XiangShan/actions/runs/9709320741/job/26802800197.

In IPrefetch:
1. s0 send r

IPrefetch: MSHR should update IPrefetch s1 waymask (#3122)

Fixes MC-Linux CI fail:
https://github.com/OpenXiangShan/XiangShan/actions/runs/9709320741/job/26802800197.

In IPrefetch:
1. s0 send read request to MetaArray
2. s1:
- receive response from MetaArray (therefore `s1_SRAM_valid === true.B`)
- and receive update request from MSHR(`fromMSHR.valid &&
!fromMSHR.bits.corrupt === true.B`)
- and `s1_fire === true.B`
3. waymasks directly from SRAM(which might be outdated) enters s2 stage,
and update request from MSHR is actually discarded.

If it is a miss(`waymask === 0.U`), IPrefetch will send miss request to
MSHR. In this case, multiple refills of the same cache block may occur,
which in turn causes a bug with multiple hits in the MetaArray.

As a fix, we should use information from MSHR to update
`s1_SRAM_waymasks` too.

Local MC-Linux test passed with seed=1244.

show more ...

ICache: implement new ICache (#3051)

Co-authored-by: xu_zh <[email protected]>

RVH: fix the errors after git rebase