Compare commits

...
1834 Commits
Author SHA1 Message Date
Ryan Houdek d3399a261b Docs: Update for release FEX-2407 2024-07-03 17:59:42 -07:00
Ryan Houdek d2437e6a21 Merge pull request #3810 from Sonicadvance1/x87_mmx_unittest
unittests: Adds MMX and x87 conflating unit test
2024-07-03 14:39:05 -07:00
Ryan Houdek 95dd6ceba8 unittests: Adds MMX and x87 conflating unit test
This failed with prior RCLSE deletion caching.
2024-07-03 13:54:07 -07:00
Alyssa Rosenzweig 1a0d135201 Merge pull request #3809 from alyssarosenzweig/rm/old-md
FEXCore: remove very out-of-date optimizer docs
2024-07-03 15:46:27 -04:00
Ryan Houdek f453e1523e Merge pull request #3803 from pmatos/NinjaCore
Use number of jobs as defined by TEST_JOB_COUNT
2024-07-03 12:42:14 -07:00
Alyssa Rosenzweig 622b0bfbc9 FEXCore: remove very out-of-date optimizer docs
most of this doesn't exist and won't exist. nothing lost here but hopes &
dreams.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-07-03 11:36:48 -04:00
Paulo Matos ad52514b97 Use number of jobs as defined by TEST_JOB_COUNT
At the moment we always run ctest with max number of cpus. If
undefined, it will keep current behaviour, otherwise it will
honour TEST_JOB_COUNT.

Therefore to run ctest one test at a time, use
`cmake ... -DTEST_JOB_COUNT=1`
2024-07-03 14:09:39 +02:00
Alyssa Rosenzweig 02a218c6e3 Merge pull request #3804 from Sonicadvance1/revert_rclse_drop
Revert removing RCLSE
2024-07-03 07:37:02 -04:00
Ryan Houdek 2d617ad173 InstcountCI: Update 2024-07-02 20:24:58 -07:00
Ryan Houdek 0d06e3e47d Revert "OpcodeDispatcher: add cache"
This reverts commit 46676ca376.
2024-07-02 20:24:57 -07:00
Ryan Houdek 78aee4d96e Revert "IR: drop RCLSE"
This reverts commit a5b24bfe4c.
2024-07-02 20:21:59 -07:00
Ryan Houdek ba04da87e5 Merge pull request #3780 from Sonicadvance1/optimize_gathers
Optimize gathers slightly
2024-07-02 10:58:38 -07:00
Ryan Houdek 2e6b08cbcb Merge pull request #3798 from Sonicadvance1/minor_128bit_vbsl_opt
Arm64: Minor VBSL optimization with SVE128
2024-07-01 18:57:46 -07:00
Ryan Houdek 472a373861 Merge pull request #3786 from Sonicadvance1/non_temporal_stores
OpcodeDispatcher: Implement support for non-temporal vector stores
2024-07-01 18:57:38 -07:00
Ryan Houdek a451420911 Merge pull request #3783 from Sonicadvance1/optimize_vector_zeroregister
OpcodeDispatcher: Optimize x86 canonical vector zero register
2024-07-01 18:57:31 -07:00
Mai 2e84f21c18 Merge pull request #3802 from Sonicadvance1/fix_sse41_helper
CodeEmitter: Fixes vector {ldr,str}{b,h} with reg-reg source
2024-07-01 20:42:49 -04:00
Ryan Houdek fb7167c2d2 CodeEmitter: Fixes vector {ldr,str}{b,h} with reg-reg source
We had failed to enable these implementations for the
`ExtendedMemOperand` helpers. We had already implemented the non-helper
forms, which are already tested in CI. These helpers just weren't
updated?

Noticed this when running libaom's SSE4.1 tests, where it managed to
execute a pmovzxbq instruction with reg+reg memory source and was
breaking the test results.

There are /very/ few vector register operations that access only 8-bit
or 16-bit in vectors so this flew under the radar for quite a while.

Fixes their unit tests.

Also adds a unittest using sse4.1 pmovzxbq to ensure we support the
reg+reg case, and also a few other instructions to test 8-bit and 16-bit
vector loads and stores.
2024-07-01 17:03:47 -07:00
Mai d884eb9287 Merge pull request #3801 from Sonicadvance1/fix_vpcmpgtw_typo
unittests: Fixes typo in vpcmpgtw test
2024-07-01 18:16:54 -04:00
Ryan Houdek 8b9b1a90e4 unittests: Fixes typo in vpcmpgtw test 2024-07-01 14:42:23 -07:00
Ryan Houdek e2d4010b59 Merge pull request #3800 from Sonicadvance1/fix_vmovlhps
AVX128: Fixes vmovlhps
2024-07-01 14:41:43 -07:00
Ryan Houdek babde31bf0 AVX128: Fixes vmovlhps
We didn't have a unit test for this and we weren't implementing it at
all.
We treated it as vmovhps/vmovhpd accidentally. Once again caught by the
libaom Intrinsics unit tests.
2024-07-01 13:54:11 -07:00
Ryan Houdek c282239077 InstcountCI: Add SVE128 VEX_map3 2024-06-30 16:27:58 -07:00
Ryan Houdek 8d28a441ab Arm64: Minor VBSL optimization with SVE128
This is a very minor performance change. On Cortex CPUs that support
SVE, they do movprfx+<instruction> fusion to remove two cycles and a
dependency from the backend.

This is a minor win to convert from ASIMD mov+bsl to SVE movprfx+bsl
because of this, saving two cycles and a dependency on Cortex A710 and
A715. This is slightly less of a win on Cortex-A720/A725 because it supports
zero-cycle vector register renames, but it is still a win on Cortex-X925
because that is an older core design that doesn't support zero-cycle
vector register renames.

Very silly little thing.
2024-06-30 16:22:29 -07:00
Ryan Houdek 5821054d91 Merge pull request #3789 from Sonicadvance1/avx128_minor_pshufb_opt
AVX128: Minor optimization to 256-bit vpshufb
2024-06-30 15:45:11 -07:00
Ryan Houdek 4626145374 Merge pull request #3792 from Sonicadvance1/avx128_fix_scalar_fma
AVX128: Fixes scalar FMA accidentally using vector wide
2024-06-30 15:36:09 -07:00
Ryan Houdek a786d3621d InstcountCI: Update for Scalar FMA 2024-06-30 14:36:56 -07:00
Ryan Houdek 1393dc2a5b AVX128: Fixes scalar FMA accidentally using vector wide 2024-06-30 14:36:33 -07:00
Ryan Houdek c4604465ba InstcountCI: Update 2024-06-30 13:41:14 -07:00
Ryan Houdek cffae9cb0f AVX128: Minor optimization to 256-bit vpshufb 2024-06-30 13:41:03 -07:00
Ryan Houdek cf24d3c33f Merge pull request #3781 from Sonicadvance1/optimize_vmovlh
AVX128: Minor optimization to vmov{l,h}{ps,pd}
2024-06-29 23:15:53 -07:00
Ryan Houdek 672e885e40 InstcountCI: Adds canonical zero register tests 2024-06-29 22:21:53 -07:00
Ryan Houdek 7d05610da7 OpcodeDispatcher: Optimize x86 canonical vector zero register
The canonical way to generate a zero register vector in x86 is to xor
itself. Capture this can convert it to canonical zero register instead.

Can get zero-cycle renamed on latest CPUs.
2024-06-29 22:21:53 -07:00
Ryan Houdek a843ecf4c8 InstcountCI: Update for non-temporal stores 2024-06-29 22:05:56 -07:00
Ryan Houdek f4ff1b0688 OpcodeDispatcher: Implement support for non-temporal vector stores
x86 doesn't have a lot of non-temporal vector stores but we do have a
few of them.

- MMX: MOVNTQ
- SSE2: MOVNTDQ, MOVNTPS, MOVNTPD
- AVX: VMOVNTDQ (128-bit & 256-bit), VMOVNTPD

Additionally SSE4a adds 32-bit and 64-bit scalar vector non-temporal
stores, which we keep as regular stores. Since ARM doesn't have matching
semantics for those.

Additionally SSE4.1 adds non-temporal vector LOADS which this doesn't
touch.
- SSE4.1: MOVNTDQA
- AVX: VMOVNTDQA (128-bit)
- AVX2: VMOVNTDQA (256-bit)

Fixes #3364
2024-06-29 22:05:56 -07:00
Ryan Houdek 2b4cec8385 Arm64: Implement support for non-temporal vector stores 2024-06-29 22:03:17 -07:00
Ryan Houdek 8ab4ab29f8 CodeEmitter: Add SVE contiguous non-temporal instructions 2024-06-29 21:51:58 -07:00
Ryan Houdek cc0509c0f3 InstcountCI: Update 2024-06-29 19:27:39 -07:00
Ryan Houdek ebfa65fedc AVX128: Minor optimization to vmov{l,h}{ps,pd} 2024-06-29 19:27:16 -07:00
Ryan Houdek a34ae24b3f InstcountCI: Update for SVE non-base address reg 2024-06-29 13:16:02 -07:00
Ryan Houdek 58ea76eb24 Arm64: Minor optimization to gather loads with no base addr register and SVE path
Arm64's SVE load instruction can be minorly optimized in the case that a
base GPR register isn't provided, as it has a version of the instruction
that doesn't require one.

The limitation of this instruction is that it doesn't support scaling at
all so it only works if the offset scale is 1.
2024-06-29 13:14:35 -07:00
Ryan Houdek e9a17b19c5 InstcountCI: Add SVE gathers without base addr 2024-06-29 13:07:32 -07:00
Ryan Houdek ce8d111453 InstcountCI: Update 2024-06-29 13:04:21 -07:00
Ryan Houdek 47fd73f6cf Arm64: Optimize non-SVE gather load
When FEX hits the optimal case that the destination isn't one of the
incoming sources (other than the incomingDest source) then we can
optimize out two moves per 128-bit lane.

Cuts 256-bit non-SVE gather loads from 50 instructions down to 46.
2024-06-29 13:02:10 -07:00
Ryan Houdek 76f3391ebc Merge pull request #3779 from Sonicadvance1/cpuinfo_cyclecounter
Linux: Calculate cycle counter frequency for cpuinfo
2024-06-29 11:58:32 -07:00
Ryan Houdek be6ff52709 Linux: Calculate cycle counter frequency for cpuinfo
Some applications don't measure rdtsc correctly and instead use cpuinfo
to get the CPU core's base clock speed. Which for most x86 CPUs is the
base clock speed which also matches their cycle counter speed.

Did this as a quick test to see if this would help `Unbound: Worlds
Apart` stuttering while BinaryNinja was disassembling the binary.

Turns out the game doesn't use cpuinfo for its cycle counter speed
determination, but it is good to implement this regardless.
2024-06-28 16:38:49 -07:00
Ryan Houdek e99e252188 Merge pull request #3731 from Sonicadvance1/avx_5
HostFeatures: Always disable AVX in 32-bit mode to protect from stack overflows
2024-06-28 13:37:55 -07:00
Ryan Houdek 98b980f7e3 TestHarnessRunner: Ensure we are still reconstructing XMM registers if we don't support AVX
Also fixes a bug where we were destroying the thread context before
reading the data from it, spooky.
2024-06-28 13:05:52 -07:00
Ryan Houdek f2f90eeb82 FEXCore: Make more distinctions between host register size and guest vector register size
We can support a few combinations of guest and host vector sizes
Host: 128-bit or 256-bit
Guest: 128-bit or 256-bit

The typical case is Host = 128-bit and Guest = 256-bit now that AVX is
implemented.
On 32-bit this changes to Host=128-bit and Guest=128-bit because we
disable AVX.

In the vixl simulator 32-bit turns in to Host=256-bit and Guest=128-bit.
And then in the vixl sim 64-bit turns in to Host=256-bit and
Guest=256-bit.

We cover all four combinations of guest and host vector register sizes!

Fixes a few assumptions that SVE256 = AVX256 basically.
2024-06-28 13:05:52 -07:00
Ryan Houdek f267fd2250 HostFeatures: Always disable AVX in 32-bit mode to protect from stack overflows 2024-06-28 13:05:52 -07:00
Ryan Houdek 500ad34769 Merge pull request #3778 from pmatos/LargeX87Blocks
Largest x87 blocks of code from games
2024-06-28 09:40:19 -07:00
Ryan Houdek 1700d54012 Merge pull request #3776 from Sonicadvance1/fix_vsib_invalid_index 2024-06-28 08:43:24 -07:00
Paulo Matos 70d8a10484 Largest x87 blocks of code from games 2024-06-28 16:50:58 +02:00
Ryan Houdek 9e94784e26 unittests: Adds test for xmm4 VSIB bug 2024-06-27 20:55:30 -07:00
Ryan Houdek 4060f4018e Frontend: Fixes invalid VSIB Index problem
In regular SIB land the index register encoding of 0b100 encodes to "no
register", this feature lets you get SIB encodings without an index
register for flexibility.

In VSIB encoding this isn't expected behaviour and instead there are no
encodings where an index register is missing. Allowing you to encode all
sixteen registers as an index register.

This was causing an abort in `AVX128_LoadVSIB` because the index turned
in to an invalid register.

Working instruction:
`vgatherdps ymm2, dword [eax+ymm5*4], ymm7`

Broken instruction:
`vgatherdps ymm0, dword [eax+ymm4*4], ymm7`

This fixes a crash in libfmod where it is using gathers in the wild.
Fixing a crash in Ender Lilies.
2024-06-27 20:55:30 -07:00
Ryan Houdek 739ac0f18f Merge pull request #3775 from Sonicadvance1/avx_bugfixes
AVX128: Some quick bugfixes
2024-06-27 17:44:12 -07:00
Ryan Houdek 98d62a7eb1 InstcountCI: Update 2024-06-27 17:21:12 -07:00
Ryan Houdek aba7a3a830 AVX128: Fixes vblendps lower and upper selector 2024-06-27 17:20:39 -07:00
Ryan Houdek 9027d1eee7 AVX128: Fixes bug in vector immediate shift 2024-06-27 16:22:14 -07:00
Ryan Houdek 4e5da4946d Merge pull request #3773 from bylaws/win-fixes
Windows: Small fixes for compat with newer toolchains/wine versions
2024-06-27 15:14:20 -07:00
Billy Laws a70e3e42b2 FEXCore: Drop unneeded MinGW library naming workaround
It's generally expected for libraries to use the .a suffix with MinGW,
and DLLs are still correctly named without the prior special handling.
2024-06-27 23:01:21 +01:00
Billy Laws 09f476924f FEXCore: Fix missing return in win32 SetSignalMask path 2024-06-27 23:01:21 +01:00
Billy Laws 230e3245fd FileLoading: Fix compilation with newer libc++ 2024-06-27 23:01:21 +01:00
Billy Laws 8de876daf2 Windows: Use newer wine unixcall API
__wine_unix_call is no longer exported in recent wine versions.
2024-06-27 23:01:19 +01:00
Ryan Houdek 53b1d155cc Merge pull request #3772 from Sonicadvance1/fix_addrsize_override
FEXCore: Fixes address size override on GPR sources and destinations
2024-06-27 15:01:08 -07:00
Ryan Houdek b0eb63ab9a FEXCore: Fixes address size override on GPR sources and destinations
When the source or destination is a register, the address size override
doesn't apply. We were accidentally applying it on all sources
regardless of type which was causing us to zero extend on operations
that aren't affected by address size override.

This fixes the OpenSSL cert error in every application, but most
importantly Steam.
2024-06-27 14:12:01 -07:00
Ryan Houdek 2e3242682d Merge pull request #3771 from alyssarosenzweig/opt/asimd-masked
OpcodeDispatcher: optimize nzcv with asimd masked load/store
2024-06-27 10:27:10 -07:00
Ryan Houdek ad4d4c9e67 Merge pull request #3770 from alyssarosenzweig/opt/vzeroall
Tiny opt for vzeroall
2024-06-27 10:25:35 -07:00
Alyssa Rosenzweig 3250d4e405 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-27 10:37:11 -04:00
Alyssa Rosenzweig 196a0531e0 OpcodeDispatcher: optimize nzcv with asimd masked load/store
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-27 10:37:06 -04:00
Alyssa Rosenzweig e61cb5b2c3 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-27 10:30:45 -04:00
Alyssa Rosenzweig f9b53c6b51 AVX_128: save a move in vzeroall
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-27 10:30:25 -04:00
Mai 58e949e148 Merge pull request #3769 from Sonicadvance1/avx2_cpuid
CPUID: Oops, forgot to enable AVX2
2024-06-26 21:17:44 -04:00
Ryan Houdek dad47b7bda CPUID: Oops, forgot to enable AVX2 2024-06-26 17:43:56 -07:00
Ryan Houdek e519bf5978 Merge pull request #3768 from Sonicadvance1/avx128_letsgo
AVX128: Enable all the things
2024-06-26 17:40:21 -07:00
Ryan Houdek fc50e52157 InstCountCI: Adds AVX128 tests 2024-06-26 16:49:00 -07:00
Ryan Houdek 7669df0e16 InstCountCI: SVE256: Fixes behaviour change 2024-06-26 16:49:00 -07:00
Ryan Houdek 4d56fec5f1 AVX128: Work around glibc fault testing 2024-06-26 16:49:00 -07:00
Ryan Houdek 8181552b16 AVX128: Actually install AVX helpers per thread.
How this didn't break the world in my testing I don't know.
2024-06-26 16:49:00 -07:00
Ryan Houdek c6c147daf6 unittests: Updates vcvtps2ph test for failure case of writing too much memory. 2024-06-26 16:49:00 -07:00
Ryan Houdek 975069825e AVX128: Fix a real bug with VCVTPS2PH 2024-06-26 16:49:00 -07:00
Ryan Houdek 5133f480d1 InstcountCI: Update for xsave/xrstor behaviour changes with AVX 2024-06-26 16:49:00 -07:00
Ryan Houdek ce4b252e5c InstCountCI: Stop disabling AVX if SVE256 is disabled. 2024-06-26 15:06:03 -07:00
Ryan Houdek 031d56de35 HostFeatures: Enables AVX unconditionally 2024-06-26 15:03:21 -07:00
Ryan Houdek 3cdaf6736b InstcountCI: Update for SVE256 FMA implementation 2024-06-26 14:56:01 -07:00
Ryan Houdek b5e696b3cb CPUID: Implement support for XCR0 when AVX is enabled
This enables AVX, AVX2, FMA3 for the entire CPUID!

```bash
$ FEX_HOSTFEATURES=enableavx,enableavx2 ./Bin/FEXInterpreter /usr/bin/cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Cortex-A78AE
stepping        : 0
microcode       : 0x0
cpu MHz         : 3000
cache size      : 512 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 12
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht tm syscall nx mmxext fxsr_opt rdtscp lm 3dnow 3dnowext constant_tsc art rep_good nopl xtoplogy nonstop_tsc cpuid tsc_known_freq pni pclmulqdq dtes64 monitor tm2 ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm cmp_legacy extapic abm 3dnowprefetc
h tce fsgsbase bmi1 avx2 smep bmi2 erms invpcid adx clflushopt clwb sha_ni clzero arat vpclmulqdq rdpid fsrm
bugs            :
bogomips        : 8000.0
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment  : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
```

Notice avx, avx2, and fma
2024-06-26 14:56:01 -07:00
Ryan Houdek 43aef377d7 HostFeatures: Allow enabling AVX without SVE256 2024-06-26 14:56:01 -07:00
Ryan Houdek add0e7a8db HostFeatures: Removes distinction between AVX and AVX2
We now no longer care about AVX versions, consolidate them in to a
single config option which enables both.
2024-06-26 14:56:01 -07:00
Ryan Houdek 52e541d453 Unittests: Stop using AVX2 flag 2024-06-26 14:56:01 -07:00
Mai a031a49546 Merge pull request #3767 from Sonicadvance1/avx128_fix_wide_shift
AVX128: Fixes wide shifts
2024-06-26 17:29:09 -04:00
Alyssa Rosenzweig 4d821b8dd8 Merge pull request #3765 from Sonicadvance1/avx128_f16c
AVX128: F16C support
2024-06-26 17:25:05 -04:00
Ryan Houdek f277025c9a AVX128: Fixes wide shifts
During refactoring this was missed and rerunning unittests locally
caught it. 256-bit operations get their shift only from the lower half
of the vector register.
2024-06-26 14:16:39 -07:00
Ryan Houdek ba28e6f82e unittests: Adds vcvtps2ph tests that use mxcsr 2024-06-26 14:08:20 -07:00
Ryan Houdek 3a89df9bed AVX128: Implement support for F16C 2024-06-26 14:05:12 -07:00
Ryan Houdek f6a0866fbb IR: Split Vector_FToF2 in to VFCVTL2 and VCVTFN2
I forgot in the narrowing case we need to be careful about insert. No IR
op used Vector_FToF2 with narrowing.
2024-06-26 14:03:41 -07:00
Ryan Houdek 756fa2ecc5 Merge pull request #3766 from alyssarosenzweig/opt/f16c-round
Optimize vcvtps2ph
2024-06-26 14:03:24 -07:00
Alyssa Rosenzweig cf834aa6da InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-26 16:46:21 -04:00
Alyssa Rosenzweig d2324f4a93 OpcodeDispatcher: optimize vcvtps2ph
We can avoid a LOT of pointless work with some dedicated IR ops for specifically
overriding the round mode.

Small behaviour change here: we no longer reset FTZ. I think this is a bug fix?
But if it's not it's not hard to fix.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-26 16:46:21 -04:00
Ryan Houdek 6226c7f4f3 Merge pull request #3757 from Sonicadvance1/avx_16
AVX128: Implement support for gathers
2024-06-26 13:29:58 -07:00
Ryan Houdek 991ecd558e InstcountCI: Update for SVE256 gathers! 2024-06-26 16:00:53 -04:00
Ryan Houdek a4fa3a460e OpcodeDispatcher: Implement AVX gathers with SVE256
Just to ensure we still have feature parity.
2024-06-26 16:00:53 -04:00
Ryan Houdek 77ba708933 AVX128: Implement support for gather load instructions
This is the last family of instructions that we needed to implement for
AVX2 to be properly advertised!
2024-06-26 16:00:53 -04:00
Ryan Houdek 662d50a966 X86Tables: Describe VPGather in the VEX tables 2024-06-26 16:00:53 -04:00
Ryan Houdek 5472d1cc04 Arm64: Implement VLoadVectorGatherMasked operation
This does a gather load three ways, SVE256, SVE128, and ASIMD.

This operation is a bit special since it it can't quite handle all
gather loadstores in the 256-bit case and requires the frontend to
decompose the operation in the case that the striding hits a mode that
SVE doesn't support!

The 128-bit case is a lot simpler since both support all the cases where
stride doesn't match. I find this to be a nice compromise while there
aren't any SVE256 products on the market.

In the 128-bit case there is an SVE path which is utilized if the passed
in stride supports what SVE understands, otherwise it falls back to an
ASIMD implementation which manually emulates everything that is
necessary.

This instruction is very explicitly doing basically exactly what AVX
gather instructions want, because it's complex enough that we don't want
to try and make this a generic solution.
2024-06-26 16:00:53 -04:00
Alyssa Rosenzweig d1d41f5645 Merge pull request #3763 from alyssarosenzweig/rclse/less-aggressive
Remove RCLSE
2024-06-26 15:14:14 -04:00
Ryan Houdek 94fd100fc7 Merge pull request #3719 from lioncash/f16c
OpcodeDispatcher: Handle F16C operations
2024-06-26 12:12:13 -07:00
Lioncache b9ff36b5d9 CPUID: Signify F16C support if AVX is available
On Aarch64 hardware, if we have SVE2 available (which we use in the AVX implementation),
then we can also enable F16C support.
2024-06-26 15:05:03 -04:00
Lioncache cd5a809ec9 OpcodeDispatcher: Handle VCVTPS2PH 2024-06-26 15:05:03 -04:00
Lioncache 045a8efbeb OpcodeDispatcher: Handle VCVTPH2PS
Fairly straightforward, since we already have handling for half-float conversions.
2024-06-26 15:05:00 -04:00
Ryan Houdek 54a1f7d833 Merge pull request #3764 from Sonicadvance1/rorx_masking
BMI2: Ensure rorx immediate masks by operation size correctly.
2024-06-26 11:52:47 -07:00
Alyssa Rosenzweig 1b496cda8f InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-26 14:49:58 -04:00
Alyssa Rosenzweig a5b24bfe4c IR: drop RCLSE
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-26 14:49:05 -04:00
Alyssa Rosenzweig 46676ca376 OpcodeDispatcher: add cache
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-26 14:49:05 -04:00
Alyssa Rosenzweig 7d939a3b3d Merge pull request #3758 from Sonicadvance1/avx_17
AVX128: FMA3
2024-06-26 14:18:32 -04:00
Ryan Houdek a515061465 BMI2: Ensure rorx immediate masks by operation size correctly. 2024-06-26 11:11:37 -07:00
Ryan Houdek 1c24d63f73 Merge pull request #3762 from alyssarosenzweig/bug/constprop-bextr 2024-06-26 09:28:18 -07:00
Alyssa Rosenzweig 7e10dba5e2 unittests: add test for a BEXTR bug
Ryan reduced this test while debugging openssl. This fails without the constprop
fix.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-26 12:06:47 -04:00
Alyssa Rosenzweig e2d73014f1 ConstProp: fix LSHR constant prop
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-26 12:06:47 -04:00
Ryan Houdek 53aa30596e InstcountCI: Update 2024-06-25 11:37:18 -07:00
Ryan Houdek 122ae5b710 unittests: Adds FMA3 unittests 2024-06-25 11:37:18 -07:00
Ryan Houdek 45c27b2965 CPUID: Enable support for FMA3 when AVX is enabled 2024-06-25 11:24:53 -07:00
Ryan Houdek 832b247fc1 SVE258: Implement support for FMA3 2024-06-25 11:24:46 -07:00
Ryan Houdek 0e8b53d566 AVX128: Implement FMA3 instructions 2024-06-25 11:23:50 -07:00
Ryan Houdek d03d69273b X86Tables: Describe FMA3 instructions 2024-06-25 11:22:27 -07:00
Ryan Houdek efa05ba19d IR: Adds support for new SUBADD FMA constants
ADDSUB didn't cover this new variant.
2024-06-25 11:22:22 -07:00
Ryan Houdek 5da205d91a Merge pull request #3760 from alyssarosenzweig/avx/vpclmulqdql
AVX128: fix VPCLMULQDQl
2024-06-25 10:31:52 -07:00
Ryan Houdek 41923bac99 OpcodeDispatcher: Fixes PCMUL with weird selectors and zero-extend
We had a bug where we weren't correctly ignoring the non-used bits in
the selector. This was causing an assert in the ARM backend.
2024-06-25 12:54:03 -04:00
Alyssa Rosenzweig c6148f6bf1 AVX128: fix VPCLMULQDQl
use the helper. I assumed the lack of zero extension here was intentional.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-25 12:51:21 -04:00
Alyssa Rosenzweig 77aaa9af4d Merge pull request #3748 from Sonicadvance1/avx_15
AVX128: More instructions Part 4
2024-06-25 12:39:48 -04:00
Ryan Houdek 00cf8d530c Merge pull request #3752 from Sonicadvance1/fma_ir_operations
ARM64: Adds new FMA vector instructions
2024-06-25 09:07:06 -07:00
Alyssa Rosenzweig 98aa58e9f5 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-25 10:03:33 -04:00
Ryan Houdek 6911917819 Disable vpclmulqdq_256 on simulator 2024-06-25 10:03:33 -04:00
Ryan Houdek a8255aa475 CPUID: Expose support for VPCLMULQDQ
Wasn't exposed before since we couldn't unit test the SVE256
implementation.
2024-06-25 10:03:33 -04:00
Ryan Houdek 48e7aae38f unittests: Adds support for 256-bit vpclmulqdq
It's easy because the test was already written for this in mind.
2024-06-25 10:03:33 -04:00
Ryan Houdek 7069643ae6 AVX128: Implement support for VPCLMULQDQ
This is just the 128-bit version twice.
2024-06-25 10:03:33 -04:00
Ryan Houdek 34272fc134 AVX128: Implement support for vperm{d,ps}! 2024-06-25 10:03:33 -04:00
Ryan Houdek 1d41002dfe AVX128: Implement support for variable vpermil{ps,pd} 2024-06-25 10:03:33 -04:00
Ryan Houdek 563bf342d5 AVX128: Implement support for vptest 2024-06-25 10:03:33 -04:00
Ryan Houdek efd5fabb95 AVX128: Implement support for vtest{ps,pd} 2024-06-25 10:03:33 -04:00
Ryan Houdek c1da525110 AVX128: Implement support for vperm2{f128,i128} 2024-06-25 10:03:33 -04:00
Ryan Houdek 5ce6c88a88 AVX128: Reenable {ldm,stm}mxcsr. Can use the regular implementation. 2024-06-25 10:03:33 -04:00
Ryan Houdek 64cce7c6fa AVX128: Implement support for xsave/xrstor 2024-06-25 10:03:33 -04:00
Ryan Houdek 4544e5b51f AVX128: Implement support for vblend{ps,pd}/vpblendvb 2024-06-25 10:03:33 -04:00
Ryan Houdek eb3e314946 AVX128: Implement support for vmaskmovdqu 2024-06-25 10:03:33 -04:00
Ryan Houdek 8b65c3de10 AVX128: Implement vmaskmov{ps,pd}, vpmaskmov{d,q} using SVE2 gather loadstores. 2024-06-25 10:03:33 -04:00
Ryan Houdek c2beb27a9d AVX128: Implement support for vpalignr 2024-06-25 10:03:33 -04:00
Ryan Houdek 05fdec9e72 AVX128: Implement support for vmpsadbw 2024-06-25 10:03:33 -04:00
Ryan Houdek e8e3c95349 AVX128: Implement support for vpsadbw 2024-06-25 10:03:33 -04:00
Ryan Houdek a87fa3f246 AVX128: Implement support for vpshufb 2024-06-25 10:03:33 -04:00
Ryan Houdek b31ad523f5 AVX128: Implement support for hsub{ps,pd} 2024-06-25 10:03:33 -04:00
Ryan Houdek 34bce540ff AVX128: Implement support for vpblendw/vpblendd/vblendps/vblendpd 2024-06-25 10:03:33 -04:00
Ryan Houdek 8ea38e1d80 AVX128: Implement support for vpmaddwd 2024-06-25 10:03:33 -04:00
Ryan Houdek ce591a9541 AVX128: Implement support for vpmaddubsw 2024-06-25 10:03:33 -04:00
Ryan Houdek a48c65cd65 AVX128: Implement support for vphaddsw 2024-06-25 10:03:33 -04:00
Ryan Houdek c283f80f48 AVX128: Implement support for vhaddpd/vphadd{w,d} 2024-06-25 10:03:33 -04:00
Ryan Houdek d6bf276b5a AVX128: Implement support for imm vpermil{ps,pd} 2024-06-25 10:03:33 -04:00
Ryan Houdek 96a51650b1 AVX128: Implement support for vshuf{ps,pd} 2024-06-25 10:03:33 -04:00
Ryan Houdek f35a9c74a2 AVX128: Implement support for vpshuf{lw,hw,d} 2024-06-25 10:03:33 -04:00
Ryan Houdek e2457943f5 AVX128: Implement support for vperm{q,pd} 2024-06-25 10:03:33 -04:00
Ryan Houdek a05644172a AVX128: Implement support for vdd{ps,pd} 2024-06-25 10:03:33 -04:00
Ryan Houdek cc168ce0fb VectorOps: Restructure DPPOpImpl. This will get reused by AVX128 2024-06-25 10:03:33 -04:00
Alyssa Rosenzweig 76bd22d279 OpcodeDispatcher: rm gratuitous lambda
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-25 10:03:33 -04:00
Alyssa Rosenzweig 18574f3cf1 OpcodeDispatcher: extract VPERMILRegOpImpl
for avx

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-25 10:03:33 -04:00
Alyssa Rosenzweig 665215ab47 OpcodeDispatcher: extract PTestOpImpl
for avx128

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-25 09:52:48 -04:00
Alyssa Rosenzweig 6009f36403 OpcodeDispatcher: extract VPERMDIndices
and rename things accordingly.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-25 09:46:17 -04:00
Alyssa Rosenzweig 2580efda0d OpcodeDispatcher: tweak VTestOp signature
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-25 09:46:17 -04:00
Ryan Houdek 3a310b8815 Merge pull request #3756 from Sonicadvance1/fix_vmovhlps
Fix VMOVLHPS instruction
2024-06-24 19:14:56 -07:00
Ryan Houdek 7ff96227c0 Merge pull request #3755 from Sonicadvance1/fix_avx128_vmovntdqa
AVX128: Fix vmovntdqa failing to zero upper 128-bits
2024-06-24 19:14:48 -07:00
Ryan Houdek 3e8d78051c InstcountCI: Update 2024-06-24 17:26:18 -07:00
Ryan Houdek bd24ebc96a unittests: Adds VMOVHLPS unit test
A bit confusing because the instruction encoding is the same between
VMOVHLPS and VMOVLPS so this unittest was missed.

Implement the test to ensure it stays working
2024-06-24 17:22:55 -07:00
Ryan Houdek 6d3745b8f1 AVX128: Fixes VMOVLHPS instruction
We didn't have unit tests for this
2024-06-24 17:22:51 -07:00
Ryan Houdek d0f0b975be SVE256: Fixes VMOVLHPS instruction
We didn't have unit tests for this
2024-06-24 17:22:47 -07:00
Ryan Houdek ff2e6ed59f X86Tables: Fixes instruction encoding for VMOVLP{S,D}
These can have both register and memory modrm encoding
2024-06-24 17:22:43 -07:00
Ryan Houdek 99b2018d0e unittests: Extend vmovntpd test 2024-06-24 16:32:13 -07:00
Ryan Houdek f0d9c8c10a AVX128: Fix vmovntdqa failing to zero upper 128-bits 2024-06-24 16:32:09 -07:00
Ryan Houdek dce1b24c00 Merge pull request #3754 from Sonicadvance1/fix_avx128_stringops
AVX128: Fixes SSE4.2 string compare instructions
2024-06-24 16:30:42 -07:00
Ryan Houdek b47e981932 AVX128: Fixes SSE4.2 string compare instructions 2024-06-24 15:54:06 -07:00
Ryan Houdek dc44eb4caf Merge pull request #3749 from Sonicadvance1/contigous_mask_optimization_removal
Arm64: Remove contiguous masked element optimization
2024-06-24 15:22:23 -07:00
Ryan Houdek dfda6733f0 Merge pull request #3750 from Sonicadvance1/pshuf_bug
OpcodeDispatcher: Fixes bug in pshuf{lw,hw}
2024-06-24 15:22:07 -07:00
Alyssa Rosenzweig 21c6986dc7 OpcodeDispatcher: tweak HSUBPOpImpl
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 18:21:08 -04:00
Alyssa Rosenzweig 635720fe12 OpcodeDispatcher: tweak PHADDSOpImpl signature
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 18:21:08 -04:00
Ryan Houdek 8c751d7423 Merge pull request #3747 from Sonicadvance1/avx_14
AVX128: More instructions Part 3
2024-06-24 12:35:22 -07:00
Ryan Houdek 702ecf7637 AVX128: Implement support for round{ss,sd} 2024-06-24 15:19:08 -04:00
Ryan Houdek 0595f1e044 AVX128: Implement support for vround{ps,pd} 2024-06-24 15:19:08 -04:00
Ryan Houdek cebb032bd3 AVX128: Implement support for vphminposuw
Reuses the non-AVX implementation since it only operates on 128-bits.
2024-06-24 15:19:08 -04:00
Ryan Houdek 8e32763ada AVX128: Implements support for AVX string ops
Reuses the implementation from the SSE4.2 implementation, just
explicitly zeroes the hardcoded YMM0's upper 128-bits.
2024-06-24 15:19:08 -04:00
Ryan Houdek 7532337231 AVX128: Implements support for vector AES instructions 2024-06-24 15:19:08 -04:00
Ryan Houdek 4a66d4570e AVX128: Implement support for a trinary operation with a passed in vector
Will be used for AES operations
2024-06-24 15:19:08 -04:00
Alyssa Rosenzweig 6f5e99d47d OpcodeDispatcher: factor out TranslateRoundType
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 15:19:08 -04:00
Alyssa Rosenzweig 9ee9f5bddd OpcodeDispatcher: tweak VectorRoundImpl signature
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 15:14:55 -04:00
Ryan Houdek ddb9f6d3ad Merge pull request #3746 from Sonicadvance1/avx_13
AVX128: More instructions
2024-06-24 11:40:52 -07:00
Ryan Houdek d29139d88a AVX128: Implement support for vextract{i,f}128 2024-06-24 14:27:19 -04:00
Ryan Houdek 317575ba99 AVX128: Implement support for cvtdq2{ps,pd} 2024-06-24 14:27:19 -04:00
Ryan Houdek d4f2638a2e AVX128: Implement support for cvt{t,}pd2pq 2024-06-24 14:27:19 -04:00
Ryan Houdek b67d9be227 AVX128: Implement support for vcvt{pd2ps,ps2pd}
Fairly complex set of instructions due to the edge cases.
2024-06-24 14:27:19 -04:00
Ryan Houdek d52add8fad AVX128: Implement support for vcvt{ss2sd,sd2ss} 2024-06-24 14:27:19 -04:00
Ryan Houdek aa9159d25c AVX128: Implement support for vpmulh{u,}w 2024-06-24 14:27:19 -04:00
Ryan Houdek 94c777259e AVX128: Implements support for vpmulhrsw 2024-06-24 14:27:19 -04:00
Ryan Houdek c9f8fa5662 AVX128: Implement support for vpmul{u,}dq 2024-06-24 14:27:19 -04:00
Ryan Houdek 64ee6b119e AVX128: Implement support for vaddsubp{s,d} 2024-06-24 14:27:19 -04:00
Ryan Houdek d2ec9a8936 AVX128: Implement support for vpsubsw 2024-06-24 14:27:19 -04:00
Ryan Houdek 2a927453f7 AVX128: Implement support for vphsub{w,d} 2024-06-24 14:27:19 -04:00
Ryan Houdek c19d489c9a AVX128: Implement support for vinsertps
This one actually reuses the core base implementation which is nice.
2024-06-24 14:27:19 -04:00
Ryan Houdek 6012eb051b AVX128: Implement support for vinsert{f128,i128} 2024-06-24 14:27:19 -04:00
Alyssa Rosenzweig 3974746473 OpcodeDispatcher: tweak PHSUBOpImpl
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 14:14:23 -04:00
Alyssa Rosenzweig e1bcdcf387 OpcodeDispatcher: tweak PHSUBSOpImpl signature
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 14:14:23 -04:00
Alyssa Rosenzweig fd5fbddae9 OpcodeDispatcher: tweak PMULLOpImpl for avx128
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 14:14:23 -04:00
Alyssa Rosenzweig 8ff72beddb OpcodeDispatcher: tweak PMULHRSWOpImpl signature for avx128
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 14:14:23 -04:00
Alyssa Rosenzweig cba5f7877b OpcodeDispatcher: tweak ADDSUBPOpImpl signature for AVX128
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 14:14:23 -04:00
Alyssa Rosenzweig 9d7e9fd9fc OpcodeDispatcher: add AVX128_Zext helper
should let us clean up a lot.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-24 14:14:23 -04:00
Ryan Houdek 082a0baff3 JIT: Implement missing Vector_FToF2 2024-06-24 14:14:23 -04:00
Ryan Houdek 3a4914315b Arm64: Remove contiguous masked element optimization
This was a premature optimization and currently breaks. Just remove it
for now.
2024-06-24 07:49:00 -07:00
Ryan Houdek 448b5a338a ARM64: Adds new FMA vector instructions 2024-06-24 07:48:05 -07:00
Ryan Houdek 9b68617fa8 InstCountCI: Update for pshuf fixes 2024-06-24 07:44:21 -07:00
Ryan Houdek 4c9890d7f8 OpcodeDispatcher: Fixes bug in pshuf{lw,hw}
This optimization was incorrect. Updates unittests to ensure it keeps
working.
2024-06-24 07:43:48 -07:00
Ryan Houdek b2db04f5d7 Merge pull request #3745 from Sonicadvance1/add_x86_cmake_assert_back
Adds back cmake error on x86-64 hosts
2024-06-24 06:44:33 -07:00
Alyssa Rosenzweig be8ff9ccb9 Merge pull request #3740 from Sonicadvance1/avx_12
AVX128: More various instructions
2024-06-24 09:28:40 -04:00
Ryan Houdek 9c531d97b0 AVX128: Implements the various vector shift instructions
These are very closely related to each other so it makes sense to
implement the roughly three different families in one commit.
2024-06-24 09:20:19 -04:00
Ryan Houdek 055d8d75a2 Revert "CI: Drop use of obsolete ENABLE_X86_HOST_DEBUG setting"
This reverts commit a054b998c5.
2024-06-24 06:05:19 -07:00
Ryan Houdek 9fcf79ce0e Adds back cmake error on x86-64 hosts 2024-06-24 06:05:19 -07:00
Ryan Houdek 6edf4619d4 Merge pull request #3742 from Sonicadvance1/export_avx_reg_helpers
FEXCore: Implement AVX reconstruction helpers
2024-06-24 05:57:44 -07:00
Ryan Houdek 8f769ce5a3 Merge pull request #3743 from alyssarosenzweig/cleanup/literal
X86Tables: add Literal() helper
2024-06-23 13:45:42 -07:00
Ryan Houdek 96ac71750a Wow64: Use SSE register reconstruction helpers
It doesn't support AVX today but it should do in the future.
2024-06-21 17:13:56 -04:00
Ryan Houdek d0852cf1bb TestHarnessRunner: Reconverge YMM registers if AVX is supported
The TestHarness infrastructure doesn't understand the difference between
converged versus split view.

So fetch the split view immediately and reconverge the view manually
inside of the state object so it continues working with the split ymm
view.
2024-06-21 17:13:56 -04:00
Ryan Houdek f5fea8af96 SignalDelegator: Use new YMM register reconstruction helpers
Otherwise we would be setting up signal handlers with incorrect register
state.
2024-06-21 17:13:56 -04:00
Ryan Houdek d52a1da501 FEXCore: Implement support for fetching/setting YMM registers
Because we have two views of the YMM registers depending on if the host
supports SVE256 or not, add helper functions to fetch them correctly.

We fetch them in the way that Linux desires them in signal handlers, if
we want to return the converged view directly, that is easy to add
support for. It's unnecessary for now.
2024-06-21 17:13:56 -04:00
Ryan Houdek abdcaa7c86 AVX128: Implement support for vpinsr{b,w,d,q} 2024-06-21 15:53:52 -04:00
Ryan Houdek ad122cf463 AVX128: Implement support for vpmovmskb 2024-06-21 15:53:52 -04:00
Ryan Houdek b58a57d225 AVX128: Implement support for vmovmskp{s,d} 2024-06-21 15:53:52 -04:00
Ryan Houdek 28d679de98 AVX128: Implement support for vpmov{s,z}{b,w,d}{w,d,q} 2024-06-21 15:53:52 -04:00
Ryan Houdek d1dd055e6a AVX128: Implement support for vpextr{b,w,d,q} 2024-06-21 15:53:52 -04:00
Ryan Houdek 3045578da4 AVX128: Implement vmov{d,q} 2024-06-21 15:53:52 -04:00
Ryan Houdek 9566dda73e AVX128: Implement support for vcmps{s,d} 2024-06-21 15:53:52 -04:00
Ryan Houdek a0ced2b685 AVX128: Implement support for vcmpp{s,d} 2024-06-21 15:50:26 -04:00
Ryan Houdek df232f567b AVX128: Implement support for v{add,sub,mul,fmin,fmax,fdiv,sqrt,rsqrt,rcp}s{s,d} 2024-06-21 15:50:05 -04:00
Ryan Houdek 2a6d6a9d13 AVX128: Implement support for v{u,}comis{s,d} 2024-06-21 15:50:05 -04:00
Alyssa Rosenzweig cd03932bd1 OpcodeDispatcher: tweak InsertScalarFCMPOpImpl signature
so AVX128 can reuse it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-21 15:50:05 -04:00
Ryan Houdek 2e5fa1ef1b Merge pull request #3739 from Sonicadvance1/avx_11
Frontend: Expose AVX W flag
2024-06-21 12:15:14 -07:00
Alyssa Rosenzweig 0c6c4cd532 OpcodeDispatcher: make FCMP more compact
I told Ryan to change this for AVX, but it needs to be changed in the original
to match!

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-21 15:02:10 -04:00
Alyssa Rosenzweig 25f8a87429 OpcodeDispatcher: use Literal() helper
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-21 14:58:49 -04:00
Alyssa Rosenzweig edf1a7970d X86Tables: add Literal() helper
Any time we get the value of Literal, we want to assert that it's actually a
literal. We've been open coding this pattern sporadically throughout the
opcodedispatcher. Let's add an ergonomic helper to fetch the value of literal,
asserting that the value is indeed literal.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-21 14:46:46 -04:00
Ryan Houdek fac9972bad Merge pull request #3741 from alyssarosenzweig/cleanup/comiss
OpcodeDispatcher: refactor Comiss helper
2024-06-21 11:43:05 -07:00
Alyssa Rosenzweig 9ecb960f3a OpcodeDispatcher: refactor Comiss helper
AVX128 will use this, it's not SSE-specific.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-21 14:23:20 -04:00
Ryan Houdek 3d26e23891 Merge pull request #3737 from Sonicadvance1/avx_10
Arm64: Implement support for emulated masked vector loadstores
2024-06-21 11:04:01 -07:00
Ryan Houdek 7bbbd95775 Merge pull request #3736 from Sonicadvance1/avx_9
AVX128: Some pun pickles, moves and conversions
2024-06-21 10:55:19 -07:00
Ryan Houdek bb308899b9 Frontend: Expose AVX W flag
Previously we could always tell the size of the operation depending on
how this effects the operating size of the instruction. Converting
64-bit down to 32-bit as an example.

AVX gather instructions are the first instruction class that can't infer
this information. The element load size is determined by the W flag but
the operating size of 128-bit or 256-bit is determined by other means.

Expose this flag so we can determine this difference. The FMA
instructions are going to need this flag as well.
2024-06-21 10:54:42 -07:00
Ryan Houdek e95c8d703c Arm64: Implement support for emulated masked vector loadstores
In order to support `vmaskmov{ps,pd}` without SVE128 this is required.
It's pretty gnarly but they aren't often used so that's fine from a
compatibility perspective.

Example SVE128 implementation:
```json
    "vmaskmovps ymm0, ymm1, [rax]": {
      "ExpectedInstructionCount": 9,
      "Comment": [
        "Map 2 0b01 0x2c 256-bit"
      ],
      "ExpectedArm64ASM": [
        "ldr q2, [x28, #32]",
        "mrs x20, nzcv",
        "cmplt p0.s, p6/z, z17.s, #0",
        "ld1w {z16.s}, p0/z, [x4]",
        "add x21, x4, #0x10 (16)",
        "cmplt p0.s, p6/z, z2.s, #0",
        "ld1w {z2.s}, p0/z, [x21]",
        "str q2, [x28, #16]",
        "msr nzcv, x20"
      ]
    },
```

Example ASIMD implementation
```json
    "vmaskmovps ymm0, ymm1, [rax]": {
      "ExpectedInstructionCount": 37,
      "Comment": [
        "Map 2 0b01 0x2c 256-bit"
      ],
      "ExpectedArm64ASM": [
        "ldr q2, [x28, #32]",
        "mrs x20, nzcv",
        "movi v0.2d, #0x0",
        "mov x1, x4",
        "mov x0, v17.d[0]",
        "tbz x0, #63, #+0x8",
        "ld1 {v0.s}[0], [x1]",
        "add x1, x1, #0x4 (4)",
        "tbz w0, #31, #+0x8",
        "ld1 {v0.s}[1], [x1]",
        "add x1, x1, #0x4 (4)",
        "mov x0, v17.d[1]",
        "tbz x0, #63, #+0x8",
        "ld1 {v0.s}[2], [x1]",
        "add x1, x1, #0x4 (4)",
        "tbz w0, #31, #+0x8",
        "ld1 {v0.s}[3], [x1]",
        "mov v16.16b, v0.16b",
        "add x21, x4, #0x10 (16)",
        "movi v0.2d, #0x0",
        "mov x1, x21",
        "mov x0, v2.d[0]",
        "tbz x0, #63, #+0x8",
        "ld1 {v0.s}[0], [x1]",
        "add x1, x1, #0x4 (4)",
        "tbz w0, #31, #+0x8",
        "ld1 {v0.s}[1], [x1]",
        "add x1, x1, #0x4 (4)",
        "mov x0, v2.d[1]",
        "tbz x0, #63, #+0x8",
        "ld1 {v0.s}[2], [x1]",
        "add x1, x1, #0x4 (4)",
        "tbz w0, #31, #+0x8",
        "ld1 {v0.s}[3], [x1]",
        "mov v2.16b, v0.16b",
        "str q2, [x28, #16]",
        "msr nzcv, x20"
      ]
    },
```

There's a little bit of an improvement where nzcv isn't needed to get
touched on the ASIMD implementation, but I'll leave that for a future
improvement.
2024-06-21 08:21:32 -07:00
Ryan Houdek 903d6a742e CPUBackend: Removes SupportsSaturatingRoundingShifts option
This has always been true ever since we removed the x86 JIT and
Interpreter. This was left over and adding more code for no reason.
2024-06-21 08:11:22 -07:00
Ryan Houdek 424218e327 AVX128: Implement support for vpsign{b,w,d} 2024-06-21 08:11:22 -07:00
Ryan Houdek 17dc03d414 AVX128: Implement support for vpack{s,u}{wb,dw} 2024-06-21 08:11:21 -07:00
Ryan Houdek baf699c6e1 AVX128: Implements support for vandnps and vpandn
This can't use the previous binary operator handler since the register
sources need to be swapped.
2024-06-21 08:11:21 -07:00
Ryan Houdek 1431af1ff5 AVX128: Implements support for vcvt{t,}s{s,d}2si 2024-06-21 08:11:21 -07:00
Ryan Houdek 775a41b903 AVX128: Implement support for vcvtsi2s{s,d} 2024-06-21 08:11:21 -07:00
Ryan Houdek 2da1e90dd5 Merge pull request #3738 from Sonicadvance1/cpuid_label
CPUID: Update labeling on some reserved bits
2024-06-21 07:41:57 -07:00
Ryan Houdek e614340c0c CPUID: Update labeling on some reserved bits
These aren't reserved and I was confused that they were missing.
2024-06-21 05:34:44 -07:00
Ryan Houdek 3c293b9aed Arm64: Loosen restrictions on V{Load,Store}VectorMasked to allow 128-bit operation 2024-06-21 04:26:09 -07:00
Ryan Houdek 283c2861c9 AVX128: Implement suppor for vlddqu 2024-06-21 00:56:36 -07:00
Ryan Houdek 757dc95116 AVX128: Implement support for the punpckh instructions 2024-06-21 00:56:32 -07:00
Ryan Houdek 6192250b8a AVX128: Implement support for the punpckl instructions 2024-06-21 00:56:28 -07:00
Ryan Houdek f489135b1d Merge pull request #3734 from Sonicadvance1/avx_8
AVX128: Move moves!
2024-06-21 00:53:41 -07:00
Ryan Houdek 4d00a52761 Merge pull request #3732 from Sonicadvance1/avx_6
unittests: Split up vtestps unittest to accumulate flags in independent registers.
2024-06-21 00:52:02 -07:00
Ryan Houdek 6941a59223 unittests: Split up vtestps unittest to accumulate flags in independent registers.
Makes it easier to see what is failing on the 128-bit side versus
256-bit side.
2024-06-21 00:45:30 -07:00
Ryan Houdek 3f232e631e Merge pull request #3730 from Sonicadvance1/avx_4
Vector: Helper refactorings
2024-06-21 00:31:14 -07:00
Ryan Houdek 6e3643c3ef Merge pull request #3714 from pmatos/FSTstiTagSet
Set tag properly in X87 FST(reg)
2024-06-21 00:27:24 -07:00
Ryan Houdek d7348c8aff Merge pull request #3683 from Sonicadvance1/fix_broken_mprotect
SMCTracking: Fix incorrect mprotect tracking
2024-06-20 22:49:51 -07:00
Ryan Houdek e7bdb8679d Merge pull request #3735 from alyssarosenzweig/instcountci/seg-reg-cases
InstCountCI: add segment register cases
2024-06-20 09:43:42 -07:00
Ryan Houdek c28824f94d AVX128: Implements support for vbroadcast* 2024-06-20 09:43:10 -07:00
Ryan Houdek 664d766b45 AVX128: Implement support for vmovshdup 2024-06-20 09:43:10 -07:00
Ryan Houdek fce694ed92 AVX128: Implement support for vmovsldup 2024-06-20 09:43:10 -07:00
Ryan Houdek 96aafb4f07 AVX128: Implement support for vmovddup
This instruction is a little weird.
When accessing memory, the 128-bit operating size of the instruction
only loads 64-bits.
Meanwhile the 256-bit operating size of the instruction fetches a full
256-bits.

Theoretically the hardware could get away with two 64-bit loads or a
wacky 24-byte load, but it looks like to simplify hardware they just
spec'd it that the 256-bit version will always load the full range.
2024-06-20 09:43:10 -07:00
Alyssa Rosenzweig a474f86ea8 InstCountCI: add segment register cases
add a bit of coverage for this funny addressing corner. We do handle this
optimally but I had to write this to check ;)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-20 11:37:35 -04:00
Ryan Houdek dbaf95a8f3 AVX128: Implement support for vmovhps/d 2024-06-20 06:53:21 -07:00
Ryan Houdek e67df96ad9 AVX128: Implement support for movlps/d 2024-06-20 06:53:17 -07:00
Ryan Houdek 56de94578d AVX128: Implement support for vmovq 2024-06-20 06:53:13 -07:00
Ryan Houdek 06fc2f5ef0 AVX128: Implement support for non-temporal moves. 2024-06-20 06:53:09 -07:00
Ryan Houdek b3ba315cbd AVX128: Implements unary/binary lambda helper 2024-06-20 06:53:05 -07:00
Ryan Houdek e5a531e683 Vector: Refactor MPSADBWOpImpl so AVX128 can use it. 2024-06-20 06:43:57 -07:00
Ryan Houdek e2de57bd04 Vector: Refactor PSADBWOpImpl so AVX128 can use it. 2024-06-20 06:43:57 -07:00
Ryan Houdek 4eebca93e3 Vector: Refactor PSHUFBOpImpl. This will be reused for AVX128 2024-06-20 06:33:27 -07:00
Ryan Houdek 3919ec9692 Vector: Expose VBLENDOpImpl in the OpcodeDispatcher. It will be reused by AVX128 2024-06-20 06:33:21 -07:00
Ryan Houdek 02aeb0ac1a Vector: Restructure PMADDWDOpImpl. It's going to get reused for AVX128 2024-06-20 06:33:15 -07:00
Ryan Houdek 206544ad09 Vector: Reconfigure PMADDUBSWOpImpl, it's going to get reused for AVX128 2024-06-20 06:33:08 -07:00
Ryan Houdek 3854cd2b2f Vector: Restruture SHUFOpImpl. AVX128 is going to reuse it. 2024-06-20 06:32:58 -07:00
Alyssa Rosenzweig b2eb8aaf66 Merge pull request #3718 from Sonicadvance1/avx128_3
OpcodeDispatcher: Adds initial groundwork for decomposed AVX operations
2024-06-20 08:57:35 -04:00
Ryan Houdek acbd920c9a OpcodeDispatcher: Adds initial groundwork for decomposed AVX operations
Only installs the tables if SVE256 isn't supported yet AVX is explicitly
enabled with HostFeatures, to protect accidental enablement early.

- Only implements 85 instructions starting out
- Basic vector moves
- Basic vector unary operations
- Basic vector binary operations
- VZeroUpper/VZeroAll

The bulk of the implementation is currently the handling for loading and
storing the halves of the registers from the context or from memory.

This means the load/store helpers must always return a pair unless only
requesting the bottom half of the register, which occurs with 128-bit
AVX operations. The store side then needing to consume the named zero
register if it occurs since those cases will zero the upper bits.

This implementation approach has a few benefits.
- I can pound this out extremely quickly
- SSE implementations are unaffected and don't need to deal with the
  insert behaviour of SVE256.
- We still keep the SVE256 implementation for the inevitable future when
  hardware vendors actually do implement it (Give it 8 years or
  something).
- We can actually unit test this path in CI once it is complete.
- We can partially optimize some paths with SVE128 (Gathers) and support
  a full ASIMD path if necessary.

One downside is that I can't enable this in CI yet because it can't pass
all unittests. but that's a non-issue since it is going to be in heavy
flux as I'm hammering out the implementation. It'll get switched on at
the end when it's passing all 1265 AVX unittests. Currently at 1001 on
this.
2024-06-20 08:44:14 -04:00
Alyssa Rosenzweig db0bdd48e5 Merge pull request #3729 from alyssarosenzweig/refactor/address-modes
OpcodeDispatcher: Refactor address modes
2024-06-20 08:18:33 -04:00
Ryan Houdek da21ee3cda Merge pull request #3692 from pmatos/AFP_RPRES_fix
Fixes AFP.NEP handling on scalar insertions
2024-06-19 19:23:49 -07:00
Ryan Houdek d2baef2b36 Merge pull request #3727 from Sonicadvance1/vaes
VAES support
2024-06-19 19:22:56 -07:00
Ryan Houdek df96bc83cc Merge pull request #3726 from Sonicadvance1/oryon_errata
HostFeatures: Work around Qualcomm Oryon RNG errata
2024-06-19 19:21:14 -07:00
Alyssa Rosenzweig ec03831a21 OpcodeDispatcher: plumb A.NonTSO deeper
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-19 08:52:07 -04:00
Alyssa Rosenzweig 9ca821316a OpcodeDispatcher: factor out DecodeAddress
this is the common guts of the load/store routines.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-19 08:52:07 -04:00
Alyssa Rosenzweig 025a060337 OpcodeDispatcher: extract IsNonTSOReg
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-19 08:52:07 -04:00
Alyssa Rosenzweig 371d6f0730 OpcodeDispatcher: extract IsOperandMem
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-19 08:52:07 -04:00
Ryan Houdek 643bc10d52 CPUID: Expose VAES if supported 2024-06-19 05:51:47 -07:00
Ryan Houdek 8fb801069f unittests: Adds new VAES tests 2024-06-19 05:51:47 -07:00
Ryan Houdek 542ed8b6ad Implement support for querying AES256 support
This is a different feature flag than regular AES as the default AES+AVX
only operates on 128-bit wide vectors.

With the newer `VAES` extension this is expanded to 256-bit.
2024-06-19 05:51:47 -07:00
Ryan Houdek 053620f4f5 Merge pull request #3728 from pmatos/PythonIgnore
Ignore python files for clang-format
2024-06-19 05:51:25 -07:00
Paulo Matos 88b01a0ca9 Ignore python files for clang-format 2024-06-19 14:23:27 +02:00
Alyssa Rosenzweig 197140498b Merge pull request #3721 from alyssarosenzweig/scripts/instcountci
Scripts: add update_instcountci.sh script
2024-06-19 06:50:00 -04:00
Paulo Matos 9acd325aa4 instcountci: Fixes AFP.NEP handling on scalar insertions 2024-06-19 10:02:54 +02:00
Paulo Matos 2483329ef6 Fixes AFP.NEP handling on scalar insertions
Fixes #3690

When doing scalar insertions, upper bits come from different arguments
depending on the operation. These are listed in the ARM spec under the
NEP bit documentation.
2024-06-19 10:02:54 +02:00
Paulo Matos f6b58b4219 instcountci: Set tag properly in X87 FST(reg) 2024-06-19 10:02:05 +02:00
Paulo Matos 6c6d86f761 unittests: Set tag properly in X87 FST(reg) 2024-06-19 10:02:05 +02:00
Paulo Matos 359221b379 Set tag properly in X87 FST(reg) 2024-06-19 10:02:05 +02:00
Ryan Houdek 87fe1d672e Merge pull request #3715 from pmatos/FXCHFlag
FXCH should set C1 to zero
2024-06-19 00:20:01 -07:00
Paulo Matos 9257221b3b instcountci: FXCH should set C1 to zero 2024-06-19 09:11:49 +02:00
Paulo Matos f9b38a1de7 FXCH should set C1 to zero 2024-06-19 08:57:48 +02:00
Ryan Houdek 67e1ac0442 Merge pull request #3725 from alyssarosenzweig/ir/vbic
IR: rename _VBic -> _VAndn
2024-06-18 16:34:26 -07:00
Ryan Houdek c57e9e008f Merge pull request #3723 from alyssarosenzweig/fexcore/zero-helper
OpcodeDispatcher: refactor zero vector loads
2024-06-18 16:34:15 -07:00
Ryan Houdek b34c23fe3d HostFeatures: Work around Qualcomm Oryon RNG errata
The Oryon is the first CPU we know of that implemented support for the
RNG extension. It also has an errata where reading the RNDRRS register
never returns success. X86's RDSEED guarantees forward progress with
enough retries.

When an x86 processor messed this up at one point, some Linux systems
would infinite loop (presumably when something in boot was filling an
entropy pool). This required a microcode change to fix that processor.

The rdseed unittest infinite loops on this platform if RNG was exposed.
2024-06-18 16:29:53 -07:00
Ryan Houdek 29f644235d Merge pull request #3724 from alyssarosenzweig/ryan-avx-cut
First few commits from Ryan's AVX branch
2024-06-18 11:39:51 -07:00
Alyssa Rosenzweig 01da5972fc IR: rename _VBic -> _VAndn
to be consistent with the scalar _Andn opcode, which is specifically named _Andn
and not _Bic.

noticed while reviewing AVX patches

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-18 14:00:01 -04:00
Alyssa Rosenzweig 643e964edd Merge pull request #3694 from Sonicadvance1/fix_3691
FEX: Consolidate JSON allocators and fix 3691
2024-06-18 13:54:10 -04:00
Ryan Houdek 30e3d795da FEX: Consolidate JSON allocators and fix 3691
Fixes #3691

We weren't checking if the file was empty before using its `at` function
member. This was causing an early crash if the config file existed but
was empty.

Consolidates the three locations that copy and pasted the json allocator
tools and adds an empty check for all of them.

Also adds two missing checks to the ThunksDB handler that could have
resulted in the same crash if ThunksDB was an empty file.
2024-06-18 13:31:25 -04:00
Alyssa Rosenzweig 89b05a2ea4 Merge pull request #3706 from Sonicadvance1/threadstateobject_cast
LinuxEmulation: Add a helper for getting the ThreadStateObject from CPU frame
2024-06-18 13:28:46 -04:00
Alyssa Rosenzweig 2e009be27c Merge pull request #3708 from Sonicadvance1/fexgetconfig_tsoemulation_facts
FEXGetConfig: Support the ability to get TSO emulation facts
2024-06-18 13:28:02 -04:00
Alyssa Rosenzweig 32150cf7b5 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-18 12:01:23 -04:00
Ryan Houdek bf812aae8f CoreState: Adds avx_high structure for tracking decoupled AVX halves.
Needed something inbetween the `InlineJITBlockHeader` and `avx_high` in
order to match alignment requirements of 16-byte for avx_high. Chose the
`DeferredSignalRefCount` because we hit it quite frequently and it is
basically the only 64-bit variable that we end up touching
significantly.

In the future the CPUState object is going to need to change its view of
the object depending on if the device supports SVE256 or not, but we
don't need to frontload the work right now. It'll become significantly
easier to support that path once the RCLSE pass gets deleted.
2024-06-18 12:00:45 -04:00
Ryan Houdek 9a71443005 CoreState: Adds a gregs offset check
This is required to be less than the maximum range for LDP and STP in
the Arm64 Dispatcher otherwise it breaks. Necessary to ensure this when
reorganizing the CoreState.
2024-06-18 12:00:45 -04:00
Ryan Houdek ee165249bc Dispatcher: Fix ARM64EC
We don't have CI for this and was missed.
2024-06-18 12:00:45 -04:00
Mai 7c7d767195 Merge pull request #3722 from alyssarosenzweig/instcountci/disable-afp
InstCountCI: explicitly disable AFP everywhere
2024-06-18 11:52:21 -04:00
Alyssa Rosenzweig af8cfb79e5 OpcodeDispatcher: refactor zero vector loads
AVX128 is going to slam this, so make it more ergonomic.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-18 11:44:46 -04:00
Alyssa Rosenzweig 27c8bf3021 InstCountCI: explicitly disable AFP everywhere
(except for when we explicitly enable AFP).

Since AFP gets saved/restored, we get `msr fpcr` garbage in random instructions
when AFP is enabled. Explicitly disable everywhere since it's not worth our time
to triage which files might hit that path. Fixes instcountci on AFP-supporting
hosts now that we have AFP enabled.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-18 11:40:20 -04:00
Alyssa Rosenzweig b0a09b31bb Scripts: add update_instcountci.sh script
This is helpful for devs working on FEXCore, I've been using this locally but it
might make sense to stick it in tree.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-18 09:07:49 -04:00
Ryan Houdek 13ebfb1a49 Merge pull request #3711 from Sonicadvance1/avx128_2
FEXCore: Disentangle the SVE256 feature from AVX
2024-06-17 17:35:15 -07:00
Ryan Houdek f863b30951 Merge pull request #3716 from alyssarosenzweig/ir-dump/unrecoverable
json_ir_generator: don't print unrecoverable temps
2024-06-17 17:25:27 -07:00
Ryan Houdek 1ce27a5e6b FEXCore: Disentangle the SVE256 feature from AVX
In quite a few locations we are mixing the case that SVE256 == AVX or
that AVX means the guest register size is 256-bit.

While this is true today, this is entanglement is going to change very
quickly and cause confusion in follow-up PRs.

Now we have SVE128, SVE256, and SVE2 HostFeatures to disambiguate the
different features which mean different things.

This PR keeps the alias that `SupportsAVX` = `SupportsSVE256 && SupportsSVE2`
but that alias is going to very quickly change its definition.
2024-06-17 17:20:32 -07:00
Ryan Houdek 933d622860 Merge pull request #3710 from Sonicadvance1/avx128_1
CoreState: Move `InlineJITBlockHeader` to the start of the struct
2024-06-17 17:17:56 -07:00
Ryan Houdek 5d67223236 Merge pull request #3707 from Sonicadvance1/clang_version_check
CMake: Add a clang version check
2024-06-17 17:17:17 -07:00
Ryan Houdek 825d2c948c Merge pull request #3717 from alyssarosenzweig/ra/ood-comment
Arm64Emitter: drop out of date comment
2024-06-17 16:02:35 -07:00
Alyssa Rosenzweig 29390b439a json_ir_generator: don't print unrecoverable temps
this makes the print more noisy for no benefit, don't do it.

before:

    %9(GPRFixed16) i32 = Add OpSize:Tmp:Size, %6(GPRFixed0) i64, %17(Invalid)
    %10(GPR0) i64 = Bfi OpSize:Tmp:Size, #0x10, #0x0, %6(GPRFixed0) i64, %9(GPRFixed16) i32
    (%11 i64) StoreRegister %6(GPRFixed0) i64, #0x11, GPR, u8:Tmp:Size
    (%12 i64) StoreRegister %9(GPRFixed16) i32, #0x10, GPR, u8:Tmp:Size
    (%13 i64) StoreRegister %10(GPR0) i64, #0x0, GPR, u8:Tmp:Size

after:

    %9(GPRFixed16) i32 = Add %6(GPRFixed0) i64, %17(Invalid)
    %10(GPR0) i64 = Bfi #0x10, #0x0, %6(GPRFixed0) i64, %9(GPRFixed16) i32
    (%11 i64) StoreRegister %6(GPRFixed0) i64, #0x11, GPR
    (%12 i64) StoreRegister %9(GPRFixed16) i32, #0x10, GPR
    (%13 i64) StoreRegister %10(GPR0) i64, #0x0, GPR

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-17 14:58:56 -04:00
Alyssa Rosenzweig 799c17eb90 Arm64Emitter: drop out of date comment
I fixed this when we landed the new RA

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-17 14:58:08 -04:00
Alyssa Rosenzweig 5fb84866e0 json_ir_generator: rework argument printing
for next commit

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-17 14:40:29 -04:00
Alyssa Rosenzweig 4965344ef5 Merge pull request #3705 from alyssarosenzweig/pre-rclse
Clean ups from my RCLSE branch
2024-06-17 14:22:01 -04:00
Alyssa Rosenzweig 46ca53ad0d Merge pull request #3704 from alyssarosenzweig/ra/spill-better
RA: priorize remat over spilling
2024-06-17 09:01:50 -04:00
Alyssa Rosenzweig 61ff1b3584 Merge pull request #3712 from alyssarosenzweig/jit/silly-assert
JIT: delete silly assert
2024-06-17 08:59:00 -04:00
Alyssa Rosenzweig 7c0c5de4bd JIT: delete silly assert
noticed in the area.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-17 08:51:22 -04:00
Ryan Houdek 8d134b8df8 InstcountCI: Update 2024-06-17 03:03:47 -07:00
Ryan Houdek a9bacc1b6b CoreState: Move InlineJITBlockHeader to the start of the struct
This currently doesn't do much but soon this will be very important to
ensure the data prefetcher of Cortex keeps the cachelines following this
variable in L1.
2024-06-17 02:59:56 -07:00
Ryan Houdek e4ff3dac86 Merge pull request #3701 from Sonicadvance1/fix_arch
FEXCore: Fixes Call with 32-bit displacement and address size override
2024-06-16 13:40:34 -07:00
Alyssa Rosenzweig 9443b18076 RegisterAllocationPass: optimize spill loop
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-16 08:15:15 -04:00
Ryan Houdek bb4e81aa19 FEXGetConfig: Support the ability to get TSO emulation facts
Allows a nice way to get information about how TSO emulation is occuring
on the individual's hardware. In particular, the more subtle details
around the implementation rather than just a hard on or off toggle.
2024-06-15 20:13:03 -07:00
Ryan Houdek a9a9f6782a CMake: Add a clang version check
Currently our minimum clang version requirement is 12.0 but soon will
require at least 13 or 14. Add a new version check in cmake to ensure
minimum version requirements.

Makes it easier to determine why a build is failing due to old compiler.
2024-06-15 18:41:19 -07:00
Ryan Houdek 2fa6c3c918 LinuxEmulation: Add a helper for getting the ThreadStateObject from CPU frame
Pulled from the seccomp WIP PR where it pulls this object more frequently.
Since it is an opaque frontend pointer it needs to be cast and we
already have a few locations that use it.

No functional change.
2024-06-15 18:31:37 -07:00
Alyssa Rosenzweig 4bd84eb523 OpcodeDispatcher: extract PF/AF invalidate helpers
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:23:47 -04:00
Alyssa Rosenzweig e2073dcd30 OpcodeDispatcher: extract safe Thunk
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:23:47 -04:00
Alyssa Rosenzweig fd72669c7e OpcodeDispatcher: extract safe Break
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:23:47 -04:00
Alyssa Rosenzweig 81c144697b OpcodeDispatcher: extract safe ExitFunction
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:23:47 -04:00
Alyssa Rosenzweig aecf180dfe OpcodeDispatcher: extract FlushRegisterCache
The "end the clause" signal. for now just flushes flags.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:23:47 -04:00
Alyssa Rosenzweig 10fa4a4f20 OpcodeDispatcher: remove never-gonna-be-done todo
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:23:47 -04:00
Alyssa Rosenzweig 534732564b OpcodeDispatcher: drop pointless thunks for packss
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:23:46 -04:00
Alyssa Rosenzweig 6a314bc9cd RegisterAllocationPass: prioritize remat over spilling
No instcountci changes yet, since nothing currently spills in instcountci. This
mitigates spilling later seen with #3703, and should help for certain
pathological blocks even without those changes (maybe we should try to get some
of those blocks in instcountci?).

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-15 20:21:57 -04:00
Ryan Houdek 1d4356b97e Change logic 2024-06-14 14:50:35 -07:00
Ryan Houdek a11566012d SMCTracking: Fix incorrect mprotect tracking
Fixes #3675

This was the first time I've ever actually dived in to this code and this
function melted my brain a bit while reading it. It was trying to be too
smart in tracking VMA splits, but if it split right at the end of a VMA
range it would then add a new range at the end where one already
existed. This then caused us to have overlapping VMA ranges and it would
completely break our SMC tracking since all tracking in the map must not
overlap.

Instead of being too smart, just break it down in to 4 merge strategies,
three of which are one shot. This is significantly easier to reason
about and each strategy is mostly self-contained. The fourth strategy in
the list is the most complex since it requires multiple steps since it
needs to walk multiple VMAs.

To help test this I added some sanity checking code that proved
invaluable to ensure everything was correct. That's not getting merged
since the overhead is too much to run, but good to have available. Diff
for this is at
  https://gist.github.com/Sonicadvance1/1ee60101ed0742b971a476fadbb51083
2024-06-14 14:36:16 -07:00
Ryan Houdek 8d929027c8 Merge pull request #3699 from Sonicadvance1/fix_3698
FEXConfig: Clear up TSO emulation string
2024-06-14 14:35:45 -07:00
Ryan Houdek 1d1ed012d8 FEXCore: Fixes Call with 32-bit displacement and address size override
FEX had a bug with this instruction where it was incorrectly using both
the address size override and operand size override to truncate the
immediate offset. This isn't how the instruction should behave as it
should actually ignore the address size override.

This now puts it correctly inline with how the jump instruction works
and adds a unit test to ensure it doesn't break again.

This fixes a crash from the Arch rootfs from the glibc dynamic linker
being compiling in a way where a call instruction was getting aligned
using this prefix (Since the compiler knew it does nothing).
2024-06-14 14:00:35 -07:00
Ryan Houdek b092b7a937 Merge pull request #3700 from lioncash/update
Externals: Update vixl submodule
2024-06-14 13:26:52 -07:00
Lioncache d133fa6dc1 ASIMD Tests: Remove erroneous disassembly tests
The vixl disassembler has gotten more strict about certain instruction types, so these tests
aren't really needed.

Alternatively, we could mark them as unallocated, but we can opt to remove them here.
2024-06-14 16:12:21 -04:00
Lioncache afa7de969e Externals: Update vixl submodule
Updates vixl to track the latest upstream changes that fix erroneous
non-zeroing behavior for 256-bit vectors
2024-06-14 15:57:03 -04:00
Ryan Houdek 41b6a89ffd FEXConfig: Clear up TSO emulation string
Fixes #3698
2024-06-14 12:39:21 -07:00
Alyssa Rosenzweig 9aa82ec5bf Merge pull request #3695 from Sonicadvance1/fix_3686
Revert "OpcodeDispatcher: optimize logical flags"
2024-06-14 07:47:51 -04:00
Ryan Houdek b17a2e9f96 Merge pull request #3697 from pmatos/ENABLEHOSTF
Use FEX_HOSTFEATURES instead of FEX_ENABLEAVX
2024-06-14 01:01:55 -07:00
Paulo Matos af4e9ceeed Remove vestigial options from workflows
FEX_ENABLEAVX was removed. Settings for host features should use
FEX_HOSTFEATURES but actually FEX_HOSTFEATURES=enableavx has different
behaviour, so we don't enable it.
2024-06-14 09:50:02 +02:00
Ryan Houdek 9c62c41f5f InstcountCI: Update 2024-06-13 19:29:55 -07:00
Ryan Houdek 184c9d21bb Revert "OpcodeDispatcher: optimize logical flags"
This reverts commit bb8336fcad.
2024-06-13 19:28:16 -07:00
Ryan Houdek 9744d8de99 Merge pull request #3689 from catfella/fix_ppa_detection
Scripts/InstallFEX: update PPA URL
2024-06-13 18:06:08 -07:00
Mikhail Nitenko f3e6ecb2c3 Scripts/InstallFEX: update PPA URL detection
Installing PPA with the script now installs a different
URL. This means that GetPPAStatus always returns false.
For the sake of backwards compatibility match the end
of the line to check.
2024-06-14 00:56:42 +00:00
Ryan Houdek aa0f2c3975 Docs: Update for release FEX-2406 2024-06-12 18:41:54 -07:00
Tony Wasserka 4dc8648d81 Merge pull request #3630 from neobrain/feature_libfwd_gl32
Library Forwarding: Add support for 32-bit OpenGL
2024-06-11 17:33:29 +02:00
Tony Wasserka 7a703e1176 Library Forwarding/GL: Enable stricter pointer parameter checks 2024-06-11 17:14:23 +02:00
Tony Wasserka 02df1a2924 Library Forwarding/GL: Avoid pointer array repacking for 64-bit guests 2024-06-11 17:14:23 +02:00
Tony Wasserka efac7efc97 Library Forwarding/GL: Remap _XDisplay returned by glXGetCurrentDisplay 2024-06-11 17:14:23 +02:00
Tony Wasserka d99b4a80c8 Library Forwarding/GL: Add glX support for 32-bit 2024-06-11 17:14:23 +02:00
Tony Wasserka 2a76744d30 Library Forwarding/GL: Add 32-bit support for most core GL APIs 2024-06-11 17:13:45 +02:00
Tony Wasserka 843b2d1969 Library Forwarding/GL: Assume void* always points to compatible data 2024-06-11 16:58:37 +02:00
Tony Wasserka b275c96889 Library Forwarding: Use the fixed-size guest type for passthrough parameters 2024-06-11 16:58:36 +02:00
Ryan Houdek 033b1ce449 Merge pull request #3688 from catfella/extend_bextr_tests
unittests/bextr: add SrcSize tests
2024-06-09 23:18:57 -07:00
Mikhail Nitenko 99a43283be unittests/bextr: add SrcSize tests
dougallj mentioned that adding these tests might expose
a bug in bextr. Since bextr implementation was changed
apparently it now works correctly, that's good.
2024-06-10 05:45:12 +00:00
Mai 55bfd6394b Merge pull request #3640 from Sonicadvance1/cleanup_execve_envp
LinuxSyscalls: Cleanup envp copying in execve
2024-06-07 21:40:13 -04:00
Ryan Houdek 14bfe6016e Merge pull request #3684 from alyssarosenzweig/constprop/cleanup
Constprop: clean up
2024-06-04 13:00:32 -07:00
Alyssa Rosenzweig 0d4ad70875 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig a8bf3859ea ConstProp: rm pointless constant folding
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig aa7dcffcea ConstProp: drop const pool heuristic
slightly worse for compile time, slightly better output, honestly I'll take the
win because this is easier to reason about.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig be1a5cea8e ConstProp: drop addressgen const pool stuff
I don't get the point, it should be handled by a combination of existing
passes/techniques just fine. no instcountci changes.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig 402ea84aa0 RedundantFlagCalculationElimination: cleanup DCE
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig 19a7b06b91 ConstProp: swallow up LongDivideElimination
as usual.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig 96bd643e5b ConstProp: always inline constants
x86/interpreter leftover, I think.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig 6b9293979c ConstProp: swallow up InlineCallOptimization
No reason to have a separate pass for this, merging should be a bit faster since
it eliminates an IR walk.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig 7d5cee4384 InlineCallOptimization: rm x86 leftover
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-04 10:09:51 -04:00
Alyssa Rosenzweig c0bab70161 Merge pull request #3682 from alyssarosenzweig/ir/ref
Find-and-replace OrderedNode* with Ref
2024-06-04 10:09:36 -04:00
Alyssa Rosenzweig 32f5a28433 IR: use Ref instead of OrderedNode
find-and-replace across the tree, excluding IR.h itself.

also excluded IRValidation because its treatment of blocks blows up and will be
reformed in the new IR anyway.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-03 12:19:34 -04:00
Alyssa Rosenzweig ce30179ed1 IR: add Ref typedef
To put new IR lipstick on the old IR pig.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-03 12:19:34 -04:00
Alyssa Rosenzweig a515b707f3 Merge pull request #3679 from Sonicadvance1/memory_model_emulation_programmer_documentation
FEXCore/docs: Adds programmer documentation about memory model emulation
2024-06-03 09:24:37 -04:00
Ryan Houdek 9ab0fa01bd Merge pull request #3681 from alyssarosenzweig/opt/shld
optimize shld
2024-06-01 11:53:42 -07:00
Alyssa Rosenzweig c3bffa2929 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 14:44:24 -04:00
Alyssa Rosenzweig 951fee361f OpcodeDispatcher: optimize shld
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 14:44:24 -04:00
Ryan Houdek 8c4860b9a7 Merge pull request #3680 from alyssarosenzweig/opt/sib 2024-06-01 11:17:54 -07:00
Ryan Houdek ee221e6a8c Syscalls: Removes unnecessary lambda that was only called once.
Local refactor.
2024-06-01 11:08:34 -07:00
Alyssa Rosenzweig f5625093bb InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:50 -04:00
Alyssa Rosenzweig abfd974d70 OpcodeDispatcher: select hardware addressing modes
Now that we have a framework to do this in.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:50 -04:00
Alyssa Rosenzweig 97966930e9 OpcodeDispatcher/x87f64: fuse addr calc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:33 -04:00
Alyssa Rosenzweig a52a2e3ae4 OpcodeDispatcher/x87: fuse addr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:33 -04:00
Alyssa Rosenzweig c49b30f105 OpcodeDispatcher/Vector: fuse addr calc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:33 -04:00
Alyssa Rosenzweig b0b4ad2083 OpcodeDispatcher: fuse xlat address
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:33 -04:00
Alyssa Rosenzweig ee4bee4fef OpcodeDispatcher: fuse BT address
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:33 -04:00
Alyssa Rosenzweig c3a0f5a2f6 OpcodeDispatcher: fuse sgdt
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:33 -04:00
Alyssa Rosenzweig 0413a6bf68 OpcodeDispatcher: improve bmi2 shift
allow upper garbage, use simpler clean.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:33 -04:00
Alyssa Rosenzweig 7bd036d1ae OpcodeDispatcher: refactor address modes
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-06-01 09:42:32 -04:00
Alyssa Rosenzweig 112c49a348 ConstProp: fix inlining shifted imm to mem instructions
hit by sse4_1-pmaxuw.c.gcc-target-test-64.jit.gcc-target-64

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-31 17:42:48 -04:00
Alyssa Rosenzweig 80878ae611 ConstProp: rework mem immediate inlining
deduplicate all the things.

functional change:
hit by sse4_1-pmaxuw.c.gcc-target-test-64.jit.gcc-target-64

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-31 17:42:48 -04:00
Alyssa Rosenzweig 85a69be5b6 ConstProp: drop address fusion
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-31 17:38:03 -04:00
Ryan Houdek 8dbfd1635a FEXCore/docs: Adds programmer documentation about memory model emulation
I keep needing to look these up to remember the limitations. Add a doc
file so I can more easily point to the information.
2024-05-31 10:36:48 -07:00
Alyssa Rosenzweig 8b5ca303e3 JIT: add asserts for invalid TSO load/store
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-31 12:12:36 -04:00
Ryan Houdek f90d2aeb6d Merge pull request #3678 from marysaka/fix/test-harness-runner-ags-segfault
Fix segfault when starting TestHarnessRunner with missing arguments
2024-05-31 08:06:54 -07:00
Mary Guillemard cd4b520f72 Fix segfault when starting TestHarnessRunner with missing arguments
Signed-off-by: Mary Guillemard <mary@mary.zone>
2024-05-31 16:54:31 +02:00
Ryan Houdek 20d5a26a72 Merge pull request #3674 from alyssarosenzweig/opt/logical-flags
Optimize logical flags
2024-05-30 12:11:21 -07:00
Alyssa Rosenzweig 9346116485 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-30 14:42:29 -04:00
Alyssa Rosenzweig bb8336fcad OpcodeDispatcher: optimize logical flags
fuse the PF write in.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-30 14:42:22 -04:00
Ryan Houdek ee96d60983 Merge pull request #3673 from alyssarosenzweig/ra/tied
Track tied sources in the IR
2024-05-30 10:55:15 -07:00
Alyssa Rosenzweig 6052b335dc Merge pull request #3666 from Sonicadvance1/fix_initial_darwinia
FileManagement: Fix fstatat/statx with self and NOFOLLOW
2024-05-29 23:11:24 -04:00
Ryan Houdek ab0a6bbe9f Merge pull request #3669 from Sonicadvance1/fix_addshift_operation
ConstProp fixes for Darwinia
2024-05-29 19:43:13 -07:00
Ryan Houdek 9dd6d8ed94 Merge pull request #3639 from Sonicadvance1/cleanupFD
FEXLoader: Cleanup FD extraction from environment variables
2024-05-29 19:18:59 -07:00
Ryan Houdek 3b5d0e3e27 FEXLoader: Cleanup FD extraction from environment variables
In preparation for seccomp execve inheritance where we need to extract
another FD from a different environment variable.

- Small function to extract the FD and also unset the environment
  variable in the same place.
   - Keeping the fetch and unset together instead of spreading to
     another location in the source.
- Extract the FD upfront instead of passing the string_view around,
  since we are unsetting the environment variable at the same place.

Future seccomp inheritance will get the FD just after the FEXFD
   - `int FEXSeccompFD {GetFEXFDFromEnv("FEX_SECCOMPFD")};`
2024-05-29 18:47:28 -07:00
Ryan Houdek 37e13cf073 FileManagement: Fix fstatat with self and NOFOLLOW
When asked to not follow the symlink, FEX needs to return data about the
symlink itself rather than following to the target executable. In that
case we need to return symlink information otherwise games that sanity
check can break.

This is what happened with Darwinia in #3662.

We return the FEXInterpreter symlink information in this case since it
doesn't return any information that is relevent to leaking emulator
state. Once the application asks to follow through to the symlink target
is when we will replace.

Also adds a unit test to ensure we don't break it.
2024-05-29 18:41:24 -07:00
Alyssa Rosenzweig 9b1b9c26cc Merge pull request #3664 from Sonicadvance1/change_timestamp
FEXLogging: Changes representation of timestamp
2024-05-29 16:44:42 -04:00
Ryan Houdek f7f3024b92 unittests/ASM: Adds SIB transpose scale register test
With a bit of pointer math it will choose the incorrect address if the
base and offset registers were transposed.
2024-05-29 11:41:20 -07:00
Alyssa Rosenzweig 11ec71a4ce InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-29 12:32:07 -04:00
Alyssa Rosenzweig 665491adf8 OpcodeDispatcher: drop weird !flagm special case
now that bfi is coalesced, this is a win.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-29 12:32:07 -04:00
Alyssa Rosenzweig 55391ccbc0 RegisterAllocationPass: try to coalesce tied sources
we'll do better in the future but this is already a win.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-29 12:32:07 -04:00
Alyssa Rosenzweig 7790d7a0b7 IR: track tied sources
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-29 12:32:07 -04:00
Ryan Houdek f3d8c2cbac unittests/ASM: Adds unittest for bug encountered in Darwinia 2024-05-29 07:29:49 -07:00
Ryan Houdek 1226069b4c InstCountCI: Update for fixes
Only prefetch hit currently since ConstProp is limited to optimizing the
ADD IROp atm.
2024-05-29 04:42:40 -07:00
Ryan Houdek 80687c8d2d ConstProp: Limits which addressing modes can be used for vector loadstores
This was causing us to generate invalid code in Darwinia, resulting in a
crash. With assertions enabled this would be picked up in the emitter.

Only implement AddShift optimizations for now because I don't want to do
the remaining optimizations in a bug fix PR.

Fixes Darwinia.
2024-05-29 04:42:11 -07:00
Ryan Houdek 920fe60492 ConstProp: Fix bug with transposed elements from AddShift op
Accidentally we were swapping which sources were the base and which was
the one getting shifted. This wasn't super common so it usually didn't
matter.

Fixes one crash in Darwinia.
2024-05-29 04:32:51 -07:00
Ryan Houdek 8c6ce2cb3b Passes/ConstProp: Have MemExtendedAddressing return a struct rather than a tuple
Makes it less confusing about which variable is the base versus the
offset.

NFC
2024-05-29 04:32:14 -07:00
Ryan Houdek 61f30d004c IR: Document AddShift behaviour
Just to clarify that Src2 is the shifted operation.
2024-05-29 04:29:09 -07:00
Ryan Houdek 95919a1ddf InstcountCI: Add addressing limit tests for base + offset<<shift
These need to be tested.
2024-05-29 04:28:24 -07:00
Ryan Houdek 35ec54f920 Merge pull request #3667 from alyssarosenzweig/opt/pcmp
Optimize PCMPESTRI flags a bit
2024-05-28 22:06:37 -07:00
Alyssa Rosenzweig 32e8a56093 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-28 09:32:14 -04:00
Alyssa Rosenzweig 136f1d0a0b OpcodeDispatcher: drop pcmpestri zext
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-28 09:32:14 -04:00
Alyssa Rosenzweig 0c042d1e85 VectorFallbacks: optimize PCMP*STRI flags
Return an NZCV.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-28 09:19:43 -04:00
Alyssa Rosenzweig ad13442be4 Merge pull request #3665 from Sonicadvance1/sse42_instcountci
InstCountCI: Adds SSE4.2 operations
2024-05-28 08:57:45 -04:00
Ryan Houdek d6b9252760 InstCountCI: Adds SSE4.2 operations
Doesn't handle all 127 combinations of the control immediate for all
four instructions. Although supplies the instruction control instruction
that A Hat in Time abuses heavily.

The SSE2 implementation of the function in vcruntime140 is likely faster
than our currently implementation but we should be able to get something
comparable. Not bad considering this is a required extension and this is
the first game we found that abuses the instruction heavily.
2024-05-28 00:55:32 -07:00
Ryan Houdek 22222ebaf5 FEXLogging: Changes representation of timestamp
This was a bit confusing to read and I had always expected to change
this at some point.

Previous:
```
[INFO][1579518391560577][1601857.1601857] clone: Unsupported flags w/o CLONE_THREAD (Shared Resources), 4100
```

Now:
```
[INFO][1590468.992593376][1629501.1629501] clone: Unsupported flags w/o CLONE_THREAD (Shared Resources), 4100
```
2024-05-27 23:36:58 -07:00
Alyssa Rosenzweig 734258e23b Merge pull request #3661 from Sonicadvance1/remove_warnings2
Removes warnings
2024-05-25 11:52:37 -04:00
Ryan Houdek 74916b3757 RAPass: Remove warnings 2024-05-24 18:41:30 -07:00
Ryan Houdek c5359264a3 VixlUtils: Remove warnings 2024-05-24 18:41:19 -07:00
Ryan Houdek 9d0ff7929e Merge pull request #3660 from alyssarosenzweig/opt/smash-less
Delete a big chunk of IR/Passes/*
2024-05-24 16:23:48 -07:00
Alyssa Rosenzweig d3eed27d17 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:45:32 -04:00
Alyssa Rosenzweig bc1669b163 DeadStoreElimination: eliminate map
use a vec. block indices will be dense in the new IR. This is memory intensive
but seems faster in practice.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig 83e417b2c6 DeadStoreElimination: combine GPR/FPR handling
slight speed up per profile.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig cb00d9171f IR: merge general DCE with flag DCE
Flag DCE needs to do general DCE anyway to converge in one pass. So we can move
the special syscall/atomic logic over to flag DCE and then drop the second DCE
pass altogether. Now local dead code of both is eliminated in a single pass.

Flag DCE is carefully written to converge in a single iteration which makes this
scheme work.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig cf77f2ae5d RedundantFlagCalculationElimination: fix convergence issue
If both the destination and the flags are dead for an AddWithFlags, we need to
eliminate it in one pass. If we only replace without elimiating, we would need a
second DCE pass to eliminate. We want DCE to finish in one pass, so fix this.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig 273d086a7b ConstProp: merge const pooling passes
walk the IR less.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig 94d9cf54bc ConstProp: don't push/pop cursor
pointless

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig 3089e0e6de ConstProp: merge masking opts with const folding
Single pass over the IR now.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig 3c088fb414 ConstProp: remove masking elimination opts
This has been deadcode since 2020. Drop it so we can focus on what *does* work
and what does matter.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Alyssa Rosenzweig 676c9a9be6 IR: do not return progress from passes
Generally, there are three reasons to track progress:

* Conditional optimizations. E.g. only run DCE if ConstProp succeeds.
* Fixed point optimizations. E.g. keep running the opt loop until convergence.
* Metadata shenianigans.

None of these apply to FEX. We explicitly do not want a nonlinear pass ordering,
instead we want just a few passes that each converge in a single iteration. We
expect them all to make progress when run. As such, tracking progress is a waste
of CPU cycles. Stop doing it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 15:44:49 -04:00
Ryan Houdek 314fea36b4 Merge pull request #3658 from Sonicadvance1/enable_afp 2024-05-24 12:23:29 -07:00
Ryan Houdek 3b7d30d26a Merge pull request #3637 from alyssarosenzweig/ra/mr 2024-05-24 12:14:17 -07:00
Alyssa Rosenzweig a8d32b9a2f InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 09:26:02 -04:00
Alyssa Rosenzweig 24cb02f4ff FEXCore: remove IRCompaction
New RA does not need it for correctness, and the slight slow down to new RA from
not compacting first is much smaller than the cost of compaction. Overall speeds
up node.js start time by ~6% on top of new RA.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 09:25:44 -04:00
Alyssa Rosenzweig 725d0e187a RegisterAllocationPass: rewrite RA
I recommend viewing the new source file as the diff is quite messy.

---

The old RA commits every "how not to write an RA" sin in the book.

Chaitin spill-one loop? Check.

Potential spilling caused by alignment issues since there's no live range
splitting? Check.

Panic spilling? Check.

Generating an interference graph with linear live ranges, so you get the code
quality of linear scan with the cost of graph colouring? Check.

...

It is wholly unsuitable to any application, and specifically unsuitable for FEX.

---

The new RA exploits a key IR invariant unique to FEX: no values are live across
block boundaries. This is validated.

Because of this invariant, all RA is block local. This lets us use a dead simple
2 pass RA that generates ~optimal code in linear time.

The first pass walks the IR backwards, analyzing the IR. This is a souped up
analogue to liveness analysis.

The second pass walks the IR forward, blasting out registers. If necessary, it
will insert spill and/or shuffle code on the fly. Spilling uses the well-known
furthest-first heuristic, which has excellent results for straight line code.

That's it :-)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 09:25:44 -04:00
Alyssa Rosenzweig d8603cb9bd RAValidation: weaken fill validation
Fails with the new RA when copies are inserted, since RAValidation loses
visibility. Nontrivial to fix, but this particular assert hopefully isn't buying
us a ton. Weaken it for now, we can revisit later.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 09:25:44 -04:00
Alyssa Rosenzweig 04c2cb5feb IR: add GPR copy/swap instructions
To implement GPRPair "properly", we need to be able to shuffle scalars around
the register file. That means we need explicit copy/swap instructions that RA
can generate them. Add some.

Swap is split in a really sketchy way, because of the 1 instruction = 1 dest
requirement. Hopefully that requirement is lifted in the future and then this
goes away.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-24 09:25:44 -04:00
Ryan Houdek 32f2decb24 Merge pull request #3656 from alyssarosenzweig/opt/asr-masking
Optimize asr
2024-05-23 21:35:48 -07:00
Ryan Houdek 6954ebe3a0 Merge pull request #3651 from Sonicadvance1/change_default_tso
Config: Change default TSO options
2024-05-23 21:35:33 -07:00
Ryan Houdek 2e40da3d6b HostFeatures: Enable AFP and RPRES
This has been investigated. Theoretically should work.
2024-05-23 13:26:05 -07:00
Alyssa Rosenzweig 559772bb03 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-23 08:27:13 -04:00
Alyssa Rosenzweig 2bbcf72e27 OpcodeDispatcher: optimize asr masking
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-23 08:27:13 -04:00
Alyssa Rosenzweig aa3a92aa60 OpcodeDispatcher: merge asr impls
so we can DRY the next patch

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-23 08:27:13 -04:00
Mai 5497240a25 Merge pull request #3655 from alyssarosenzweig/opt/wacky-imul
Optimize large sign-extended constants
2024-05-22 23:27:09 -04:00
Alyssa Rosenzweig 79c609d0f5 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-22 22:53:19 -04:00
Alyssa Rosenzweig b33788e765 Arm64Emitter: handle another class of constants
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-22 22:50:20 -04:00
Alyssa Rosenzweig 8340012466 InstCountCI: add wacky imul
found in bytemark, exposes an interesting constant case

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-22 22:50:19 -04:00
Ryan Houdek 0adcc779cf Merge pull request #3654 from alyssarosenzweig/opt/movsx
Optimize sign-extension
2024-05-22 19:29:51 -07:00
Alyssa Rosenzweig 9d0718fbc4 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-22 21:00:25 -04:00
Alyssa Rosenzweig 53567a6526 OpcodeDispatcher: optimize movsxd
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-22 21:00:25 -04:00
Alyssa Rosenzweig 90fb5f038b OpcodeDispatcher: optimize movsx
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-22 21:00:25 -04:00
Mai 063b1eb936 Merge pull request #3652 from alyssarosenzweig/instcountci/more-bytemark
InstructionCountCI: add bytemark hot block
2024-05-22 14:20:13 -04:00
Alyssa Rosenzweig 9f243c8f7b InstructionCountCI: add bytemark hot block
this was basically practice for using perf with FEX, but a few things do stand
out in the assembly as suboptimal.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-22 14:09:50 -04:00
Ryan Houdek 2dc600f283 Config: Change default TSO options
After two months of testing I finally have enough confidence that these
default options getting changed is safe enough that most games won't
notice the difference. But the performance differences can be wild.

Might want to come back and reenable this on platforms that support
hardware TSO and LRCPC3 (for the vector feature), but we can care once
those platforms come up to speed.
2024-05-21 18:08:58 -07:00
Ryan Houdek a01402d502 Merge pull request #3650 from alyssarosenzweig/unittests/xess
unittests: add XeSS test
2024-05-21 17:15:08 -07:00
Ryan Houdek c90036aeea Merge pull request #3649 from alyssarosenzweig/ra/validate-less-hard
Simplify/fix our validation passes
2024-05-21 17:12:49 -07:00
Alyssa Rosenzweig dfb751eea0 unittests: add XeSS test
Useful smoke test for the asymptoptic behaviour of our constant pooling code.
Not useful for correctness testing, so skip in CI.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 19:41:47 -04:00
Alyssa Rosenzweig 6e0f5eccb3 RAValidation: fix spillregister validation
It doesn't write to its node. Fixes spurious

  %7: Arg[0] expects reg0 to contain %4, but it actually contains %16

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 19:34:31 -04:00
Alyssa Rosenzweig 579fb42458 RAValidation: allow filling the same slot multiple times
This can be a reasonable thing to do!

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 19:34:31 -04:00
Alyssa Rosenzweig f4b487352c RAValidation: defeature control flow analysis
now that we've eliminated cross block liveness, we can do our validation locally
too for a massive simplification.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 19:34:31 -04:00
Alyssa Rosenzweig 4448f84f29 IRValidation: merge in ValueDominanceValidation
All we actually need to validate is that each source has been previously defined
within the block. That checks everything we care about now.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 19:34:31 -04:00
Alyssa Rosenzweig 9e1e602e09 BitSet: fix memset/memclear logic
Missing a factored of 4, causing a buffer overflow.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 19:32:54 -04:00
Ryan Houdek ca70e387ec Merge pull request #3648 from alyssarosenzweig/ra/pair-extract
Slightly improve pair coalescing + memcpy fix from RA branch
2024-05-21 16:19:35 -07:00
Ryan Houdek 9a483107e3 Merge pull request #3647 from alyssarosenzweig/ir/pass-simpler
ConstProp, RCLSE: simplifications
2024-05-21 16:11:36 -07:00
Ryan Houdek 3bac767866 Merge pull request #3645 from neobrain/refactor_aotir
AOTIR: Refactor interfaces to clarify ownership flow
2024-05-21 16:03:06 -07:00
Ryan Houdek 7b4e48480b Merge pull request #3646 from alyssarosenzweig/opt/minor-disp
OpcodeDispatcher: eliminate some Bfe's
2024-05-21 15:57:52 -07:00
Alyssa Rosenzweig 101bba4808 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 17:14:57 -04:00
Alyssa Rosenzweig a31c3c1c15 OpcodeDispatcher: use ExtractPair
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 17:14:57 -04:00
Alyssa Rosenzweig 1a18e392f8 OpcodeDispatcher: add ExtractPair helper
terser and will aid coalescing, as well as eventual transition to multidest
extracts which is what we'll actually want.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 17:14:57 -04:00
Alyssa Rosenzweig 4c7595c68a JIT: allow coalescing ExtractElementPair
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 17:14:57 -04:00
Alyssa Rosenzweig 1a467f0ebd IR: reduce memcpy worstcase reg pressure
This avoids a bunch of sharp edges for RA at a small cost when obscure
segment registers are used.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:54:06 -04:00
Alyssa Rosenzweig 06e7360f4c RCLSE: only run once, do not DCE
From a theoretical perspective, we should not need to run RCLSE more than once.
If there are convergence issues with the current implementation, they should be
fixed instead of bandaged around. Fortunately, this has no instcountci changes.

Brings RCLSE cost down from like 12% to 5%.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:46:44 -04:00
Alyssa Rosenzweig ec3b72e17e ConstProp: drop select folding
no instcountci changes.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:46:44 -04:00
Alyssa Rosenzweig 259e1b75a4 ConstProp: rm printfs
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:46:44 -04:00
Alyssa Rosenzweig f9642cba7a ConstProp: rm pointless opcode casts
Just use the headers directly

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:46:44 -04:00
Alyssa Rosenzweig c01c415030 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:40:59 -04:00
Alyssa Rosenzweig 50e56358c3 OpcodeDispatcher: eliminate Bfe's with cmpxchg
ConstProp was catching these but they're pointless.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:39:00 -04:00
Alyssa Rosenzweig 465dbc260f OpcodeDispatcher: eliminate Bfe's with lea
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-21 16:39:00 -04:00
Tony Wasserka 9f291f3adb AOTIR: Remove obsolete fields from serialized data 2024-05-21 17:54:28 +02:00
Tony Wasserka f2acc3da4c Core: Clean up ownership management for IRListView and RegisterAllocationData
IRListView is now purely a view type. Instead, ownership is managed on-demand
by a separate interface (IRStorageBase). Materialization of IRListViews to
owning types is moved to this interface as well.

This also avoids unneeded copies of the data.
2024-05-21 17:51:17 +02:00
Tony Wasserka f8f165d96d AOTIR: Remove redundant variable declarations
This code was already using structured bindings anyway and just reassigned
the values to different variables.
2024-05-21 17:38:41 +02:00
Tony Wasserka baef95992c AOTIR: Drop unneeded local variable 2024-05-21 17:38:41 +02:00
Tony Wasserka 97e18c8469 AOTIR: Drop effectively unused parameter from PreGenerateIRFetch 2024-05-21 17:38:41 +02:00
Tony Wasserka 6e04f7368b AOT: Use std::optional to replace a validity boolean in PreGenerateIRFetch 2024-05-21 17:38:41 +02:00
Tony Wasserka 55a835ebb8 AOTIR: Clarify serialization code
The comments weren't too helpful. Using the struct types directly conveys the
same information more clearly.
2024-05-21 17:38:41 +02:00
Alyssa Rosenzweig 85776c2537 Merge pull request #3643 from alyssarosenzweig/opt/shift-garbage
Allow garbage on more shifts
2024-05-21 11:05:47 -04:00
Ryan Houdek e3ec25d9db Merge pull request #3638 from alyssarosenzweig/jit/dedupe-vec
JIT/VectorOps: deduplicate common implementations
2024-05-20 14:44:52 -07:00
Alyssa Rosenzweig ebfcc1e835 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-20 10:33:15 -04:00
Alyssa Rosenzweig 769b2c2a46 OpcodeDispatcher: allow garbage on shift dests
doesn't matter for left shifts (we mask off the garbage), or 32-bit shifts, or
shifts where we explicitly sbfe after.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-20 10:33:15 -04:00
Alyssa Rosenzweig 3b2100307e OpcodeDispatcher: allow garbage on more shifts
we're masking anyway

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-20 10:33:15 -04:00
Ryan Houdek 28cc179214 LinuxSyscalls: Cleanup envp copying in execve
In preparation for seccomp execve inheritance.

We are going to need to add a new environment variable earlier in the
execve sequence to handle inheritance in the case of binfmt_misc.

No functional change in regards to envp handling.

Minor change around execveat with FD without binfmt_misc. In the case
that execveat returned an error and we did a `dup` of the FD then we
would have an FD leak. Make sure to close the duplicated FD in that
instance.
2024-05-20 07:27:37 -07:00
Alyssa Rosenzweig 3c0f243a2d JIT: dedupe MapCC
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-20 10:25:56 -04:00
Alyssa Rosenzweig bbf1563f80 JIT/ALUOps: extract DEF_BINOP_WITH_CONSTANT
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-20 10:25:56 -04:00
Alyssa Rosenzweig ed6b1011f9 JIT/VectorOps: deduplicate common implementations
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-20 10:25:55 -04:00
Ryan Houdek e3e7f0279c Merge pull request #3644 from alyssarosenzweig/clang-format/left
clang-format: left-align escaped newlines
2024-05-20 07:12:50 -07:00
Alyssa Rosenzweig a10f984b1c clang-format: left-align escaped newlines
alternative to #3638. this is theoretically better for side-by-side diffs. in
practice it may make other diffs worse since all the \'s change when part of the
macro change.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-20 09:47:21 -04:00
Ryan Houdek 663f3d8b5a Merge pull request #3641 from Sonicadvance1/instcountci_flake
InstCountCI: Hardcode the offset to load tests into
2024-05-20 06:45:47 -07:00
Ryan Houdek b1f7be2f6c InstCountCI: Update 2024-05-18 18:08:38 -07:00
Ryan Houdek b83cbcb33c ConstProp: Bandage fix for instcountci
Fixed offset x86 code doesn't quite solve the issue, so adjust this
heuristic just to get instcounci to stop flaking.

This code is going to heavily change soon anyway so +50 doesn't change
much.
2024-05-18 18:06:25 -07:00
Ryan Houdek ac1a096bae InstCountCI: Hardcode the offset to load tests into
Depending on where the assembly was getting loaded in to memory it was
causing slight code generation differences.

Map the entire file to the same fixed offset as our ASM tests to ensure
consistency and removing flakes in CI.
2024-05-18 17:00:28 -07:00
Ryan Houdek 048c8ded88 Merge pull request #3622 from Sonicadvance1/move_emitter 2024-05-17 10:41:51 -07:00
Alyssa Rosenzweig 948938bf4b Merge pull request #3636 from alyssarosenzweig/jit/factor-vec
JIT: factor out sub reg size conversion
2024-05-17 09:40:32 -04:00
Alyssa Rosenzweig bb064c7334 JIT/AtomicOps: factor out elementsize
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 17:28:58 -04:00
Alyssa Rosenzweig 2d3d49b900 JIT/ConversionOps: use ConvertSubRegSize*
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 17:18:59 -04:00
Alyssa Rosenzweig ea7096ed5b JIT/MemoryOps: use ConvertSubRegSize8
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 17:16:12 -04:00
Alyssa Rosenzweig 7a0f6c0a80 JIT: factor ConvertSize helper
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 17:13:00 -04:00
Alyssa Rosenzweig e4ee35a925 JIT: factor out sub reg size conversion
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 15:56:05 -04:00
Ryan Houdek d3ab9bdef6 Remove Float16
We aren't using it. We won't be using it. We need unit tests in our
lives if we want this.
2024-05-16 12:06:54 -07:00
Ryan Houdek 926eefc86c Merge pull request #3635 from alyssarosenzweig/opt/flag-store
OpcodeDispatcher: reorder some moves
2024-05-16 10:58:40 -07:00
Ryan Houdek 3eb7a5b998 Merge pull request #3632 from pmatos/RemoveBlocks
Use erase-remove idiom to remove element
2024-05-16 10:50:50 -07:00
Ryan Houdek 58614ff131 Merge pull request #3634 from alyssarosenzweig/constprop/leftover
ConstProp: remove x86 jit leftover
2024-05-16 10:48:31 -07:00
Alyssa Rosenzweig bf3a09e5e3 ConstProp: remove x86 jit leftover
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 09:21:41 -04:00
Alyssa Rosenzweig 83c536c47f InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 08:32:11 -04:00
Alyssa Rosenzweig 7b39e57e72 OpcodeDispatcher: defer overwritten store
this can save moves, as it's a bit easier to reason about the live ranges.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-16 08:31:58 -04:00
Alyssa Rosenzweig 5bedf32666 Merge pull request #3633 from pmatos/UndefShift
Fix left shift undefined behaviour
2024-05-16 07:45:38 -04:00
Paulo Matos 1eb7be9870 Check BitOffset instead of zeroext 2024-05-16 10:50:40 +02:00
Paulo Matos 93e4288c57 Fix left shift undefined behaviour
Here BitOffset can have values higher than 32.
2024-05-16 10:22:03 +02:00
Paulo Matos 9ca4868833 Use erase-remove idiom to remove element
Fixes #3631
2024-05-16 10:16:01 +02:00
Alyssa Rosenzweig c5f8ea58e9 Merge pull request #3629 from pmatos/Unsup-typo
NFC: Fix typo
2024-05-15 10:31:38 -04:00
Paulo Matos 5bee17bee1 NFC: Fix typo 2024-05-15 15:10:00 +02:00
Ryan Houdek efe7c54374 Merge pull request #3625 from Sonicadvance1/restricted_inst
FEXCore: Fixes the difference between CPL-0 and undefined instructions
2024-05-14 07:11:19 -07:00
Ryan Houdek 9e1840e974 FEXCore: Moves CodeEmitter to FHU
Now that the vixl dependency is gone, this gets moved to FHU since the
frontend is going to need it for a microjit.
2024-05-13 12:48:10 -07:00
Ryan Houdek 1f40590f9a Emitter: Inline IsImmLogical from vixl
The only core vixl usage we use in the emitter. Is a complete pain to
reimplement so keep it around.
2024-05-13 12:48:10 -07:00
Ryan Houdek 64a3bc235d unittests/Emitter: Ensures coverage of imm float encodings
To ensure we round everything correctly for the new float16 class
2024-05-13 12:48:10 -07:00
Ryan Houdek 7d9af246ea CodeEmitter: Removes vixl Float16 usage
Creating local Float16 helper which handles our needs
2024-05-13 12:48:09 -07:00
Ryan Houdek 6d3471bcaa Merge pull request #3627 from Sonicadvance1/timeout_merge_base
Github: Support a timeout on checkout
2024-05-13 11:53:11 -07:00
Ryan Houdek a8714dbd49 Github: Support a timeout on checkout
Sometimes github or the CI runner times out trying to checkout the
source and stays timing out forever.

Give it a three minute timeout otherwise the CI runner will stall
forever.
2024-05-13 11:26:54 -07:00
Ryan Houdek 512312fa06 FEXLinuxTests: Implements a test for the new instructions 2024-05-13 11:12:26 -07:00
Ryan Houdek 010028e381 FEXCore: Fixes the difference between CPL-0 and undefined instructions
undefined instructions are expected to return SIGILL, while implemented
instructions that aren't available in CPL-3 are expected to SIGSEGV.

Noticed this while testing out CPU-Z, it installs a kernel module and
does a bunch of `RDMSR` and `OUTS` instructions. Decided to walk through
the rest of the instructions in the `System Instruction Reference`
section.

Turns out there's a bunch of oddities in there that we don't support.
First step is to go through all the explicitl SIGILL and SIGSEGV and
implement a test for them.

Next step will be implementing the remaining operations that are
considered "System" operations but are still available in CPL-3.
This list includes:
- lar
- lgdt
- lsl
- sidt
- sldt
- stac
- clac
- verr
- verw
2024-05-13 11:12:26 -07:00
Ryan Houdek f27f1871e4 Merge pull request #3624 from Sonicadvance1/faulty_mc_fault_face
FEXCore: Get rid of DeferredSignalFaultAddress and use the InterruptFaultPage
2024-05-13 03:22:43 -07:00
Ryan Houdek 3a7aa83ab1 Merge pull request #3626 from pmatos/TestClangIgnore
Fix exec path where file needs to be ignored
2024-05-13 02:55:33 -07:00
Paulo Matos bcc136c3b9 Fix exec path where file needs to be ignored
Ignored files were not being checked. Both clang-format.py wrapper
and code-format-helper where not aligned.
2024-05-13 11:41:44 +02:00
Ryan Houdek 3da31830d1 InstcountCI: Update 2024-05-10 15:34:13 -07:00
Ryan Houdek d19b57a52e FEXCore: Get rid of DeferredSignalFaultAddress and use the InterruptFaultPage
Arm64ec introduced the InterruptFaultPage which is lower overhead since
instead of ldr+str it just turns in to a single str. We were already
allocating the space, FEXCore and the frontend signal delegator just
needed to be updated to understand the new location.

We can additionally use this in the future if we want to make deferred
async signals INSIDE the JIT only cost a single str as well.
2024-05-10 15:31:28 -07:00
Ryan Houdek ef6d640a8c Merge pull request #3612 from Sonicadvance1/threadmanager_move
FEXLoader: Changes frontend thread management to wrap FEXCore thread objects
2024-05-09 09:27:11 -07:00
Ryan Houdek 2cae2f2462 Merge pull request #3617 from bylaws/arm64ec-dispatcher
FEXCore: ARM64EC x64 entry/exit support
2024-05-08 12:25:26 -07:00
Ryan Houdek 1fde5d7fca Merge pull request #3621 from alyssarosenzweig/ra/drop-avx
RegisterAllocationPass: drop AVX flag
2024-05-08 11:42:53 -07:00
Ryan Houdek 10de2f83ac Merge pull request #3620 from alyssarosenzweig/ir/burn-parser
IR: drop IRParser
2024-05-08 11:29:37 -07:00
Alyssa Rosenzweig 9d86e11a47 RegisterAllocationPass: drop AVX flag
RA should not depend on whether we support AVX, that's a huge layering
violation! and fortunately, it does not.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:26:31 -04:00
Alyssa Rosenzweig 4d503d3155 RegisterAllocationPass: drop prewritable check
always true.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:24:41 -04:00
Alyssa Rosenzweig 7e663b91df IR: drop IRParser
Aside from its own self-test, the parser is unused and should remain that way,
since it's a maintenance burden with no real benefit. Burn it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:16:54 -04:00
Alyssa Rosenzweig 47242dc190 Merge pull request #3616 from alyssarosenzweig/sra/simplify-1
SRA controlled burn
2024-05-08 14:10:48 -04:00
Alyssa Rosenzweig e13c8e3295 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 3c3ba62c10 MemoryOps: optimize 32-bit SRA case
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 34fe56dfb2 DeadStoreElimination: CSE block info
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig ecf8cde5e0 DeadStoreElimination: group common logic
slightly less obnoxious copypaste.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 55284aad7e DeadStoreElimination: don't handle partial stores
SRA replaces the whole contents of the destination.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 3afc35f7b4 DeadStoreElimination: simplify
use registers internally, not synthesized offsets

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 1058428a51 IR: document invariant on SRA
This lets us simplify a lot!

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig a2fc51fc7b IR: specify registers, not offsets for SRA
SRA is fundamentally about hardware registers, not stores into a
software-defined context. So, it should take a register instead of an offset.
This makes all the unaligned special cases unrepresentable (by design).

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 1848629ba5 RegisterAllocationPass: drop aliasable check
always true with the new ir invariants.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 3399577330 JIT: clean up fpr sra
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 18bfc8afd0 JIT: clean up gpr sra handling
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 76b023ed3e JIT: drop unaligned and partial SRA handling
This is all dead, assert as much so it stays that way.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig b91b0e9d65 IR: infer SRA static class
no need to stick it in the IR.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Alyssa Rosenzweig 74489a4177 IR: remove dead SRA flags
I don't know what these were meant for, and I don't care (-:

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-05-08 14:01:42 -04:00
Ryan Houdek 55d1d6bcd4 Merge pull request #3615 from bylaws/wow64-fix
Fix WOW64 frontend with recent wine versions
2024-05-07 22:35:32 -07:00
Ryan Houdek cd249e2c3a Merge pull request #3614 from Sonicadvance1/remove_temporary_allocation
FEXServer: Removes temporary variable allocation
2024-05-07 22:35:23 -07:00
Billy Laws 61cd835754 Update InstCountCI 2024-05-06 17:37:43 +00:00
Billy Laws bd24364c1b FEXCore: Switch stacks before exiting the JIT on ARM64EC
This removes the need for the frontend to have any knowledge of FEX's
SRA layout.
2024-05-06 15:41:34 +00:00
Billy Laws ab516d7b79 Dispatcher: Implement ARM64EC SRA setup entrypoints
While the ARM64EC ABI mostly matches FEX's SRA, the stack still needs to
be switched to the emulator stack and target RIP stored into the FEX
context before jumping to the dispatcher loop.
2024-05-06 15:41:34 +00:00
Billy Laws d25ed4b0bf Dispatcher: Block system call callbacks when compiling code
These callbacks are used for code invalidation and setting the right
emulated CPU features, neither of which are necessary for syscalls made
from within FEX. Avoid calling them to prevent deadlocks caused by
nested locks during compilation.
2024-05-06 15:41:28 +00:00
Billy Laws c521d2b48d WOW64: Support unwinding past FEX from within syscall handlers
This is required by recent wine changes to use longjmp for user
callbacks. Switch to saving the context at every simulate call and
setting the unwind SP/PC to that context with a small SEH trampoline
for the syscall handler.
2024-05-06 15:26:36 +00:00
Billy Laws 9ed8165405 WOW64: Dynamically allocate unixcall/syscall entrypoints
Removes the requirement that FEX needs to be loaded as part of the lower
32-bit address space.
2024-05-06 14:55:59 +00:00
Ryan Houdek 5099b2b5dc FEXServer: Removes temporary variable allocation
Was causing unnecessary memory allocation churn when a FEXInterpreter
was asking for the rootfs folder path.
2024-05-05 14:11:26 -07:00
Ryan Houdek d372552593 FEXLoader: Changes frontend thread management to wrap FEXCore thread objects
A bit of refactoring necessary before we can move the remaining Linux
specific code to the frontend.

Most of this taken from #3535 but attempting to be NFC as much as
possible.
2024-05-05 07:43:09 -07:00
Ryan Houdek 729e32ccc2 Linux: Move ThreadManager to its own header 2024-05-05 06:32:59 -07:00
Mai 170204d6f1 Merge pull request #3609 from neobrain/fix_catch2_setting
CMake: Remove obsolete Catch2 setting
2024-05-04 00:04:52 -04:00
Mai f7bfecd3f1 Merge pull request #3610 from Sonicadvance1/support_oryon_named
CPUID: Adds Qualcomm Oryon product name
2024-05-04 00:04:29 -04:00
Ryan Houdek 5f0427c253 CPUID: Adds Qualcomm Oryon product name
From https://github.com/llvm/llvm-project/pull/91022

Easy enough
2024-05-03 20:16:46 -07:00
Tony Wasserka 472860a840 CMake: Remove obsolete Catch2 setting 2024-05-03 15:25:42 +02:00
Mai eddb7d12cc Merge pull request #3608 from Sonicadvance1/readme_remove_x86
Readme: Remove misleading text about x86 hosts being supported
2024-05-03 00:04:53 -04:00
Ryan Houdek 789a9f19c0 Readme: Remove misleading text about x86 hosts being supported
This is no longer the case as x86-64 hosts is purely a development
vehicle and it is not expected for users to try this.
2024-05-02 20:38:13 -07:00
Ryan Houdek f70aafb211 Merge pull request #3607 from teohhanhui/fix/open-mode
Pass compulsory `mode` argument to `open` when `O_CREAT` is used
2024-05-02 13:08:47 -07:00
Teoh Han Hui 7519af2819 Pass compulsory mode argument to open when O_CREAT is used
From `man 2 open`:

> The mode argument must be supplied if O_CREAT or O_TMPFILE is
> specified in flags; if it is not supplied, some arbitrary bytes
> from the stack will be applied as the file mode.
2024-05-03 03:16:29 +08:00
Ryan Houdek 7c79e5dea1 Docs: Update for release FEX-2405 2024-05-02 11:35:14 -07:00
Ryan Houdek faa494c288 Merge pull request #3605 from Sonicadvance1/move_fex_versionstring_cpuid
CPUID: Removes FEX version string from CPU model name
2024-05-02 11:20:49 -07:00
Ryan Houdek b33e0e3839 Merge pull request #3584 from neobrain/feature_libfwd_guestx11
Library Forwarding: Support libGL/libvulkan without forwarding libX11
2024-05-02 11:20:34 -07:00
Tony Wasserka a15ed4c9da Library Forwarding: Drop support for libX11
The implementation of this has been brittle and is architecturally
incompatible with 32-bit guests. It's unlikely this could be fixed with
incremental improvements.

Since libGL and libvulkan can be forwarded independently of libX11 now,
these libX11 bits can be dropped without negative impact on compatibility.
2024-05-02 20:02:02 +02:00
Tony Wasserka 4b76eb0b3f Library Forwarding/vulkan: Reload pointers for extension functions more aggressively
Some applications create multiple Vulkan instances with different sets of
extensions. We might hence miss some of these pointers during the initial
function pointer query.
2024-05-02 20:02:02 +02:00
Tony Wasserka 86315027c3 Library Forwarding: Support Vulkan forwarding with guest-libX11 2024-05-02 18:06:54 +02:00
Tony Wasserka a114850c2a Library Forwarding: Support GL forwarding with guest-libX11 2024-05-02 17:59:28 +02:00
Tony Wasserka 997d1bf04b Library Forwarding: Add helper class for interception X11 communication
X11 displays and xcb connections managed by the guest libX11 can't be used by
the host, but we can create intermediary objects using the host libX11. This
allows to connect guest-managed objects to the host window system integration
APIs in OpenGL/Vulkan.
2024-05-02 17:59:28 +02:00
Tony Wasserka c294782a60 Library Forwarding: Support annotating function return types 2024-05-02 17:59:28 +02:00
Mai 9781b957d0 Merge pull request #3606 from Sonicadvance1/fix_inverted_rdtscp
CPUID: Fix inverted RDTSCP check
2024-05-01 22:21:09 -04:00
Ryan Houdek 6228226c08 CPUID: Fix inverted RDTSCP check
This was inverted and always enabling the RDTSCP cpuid bit for wine.
Thus always disabling it elsewhere.
2024-05-01 18:31:41 -07:00
Ryan Houdek 31341bb7c2 CPUID: Removes FEX version string from CPU model name
Moves it to the hypervisor leafs.

Before:
```bash
$ FEXBash 'cat /proc/cpuinfo | grep "model name"'
model name      : FEX-2404-101-gf9effcb           Cortex-A78C
model name      : FEX-2404-101-gf9effcb           Cortex-A78C
model name      : FEX-2404-101-gf9effcb           Cortex-A78C
model name      : FEX-2404-101-gf9effcb           Cortex-A78C
model name      : FEX-2404-101-gf9effcb           Cortex-X1C
model name      : FEX-2404-101-gf9effcb           Cortex-X1C
model name      : FEX-2404-101-gf9effcb           Cortex-X1C
model name      : FEX-2404-101-gf9effcb           Cortex-X1C
```

After:
```bash
$ FEXBash 'cat /proc/cpuinfo | grep "model name"'
model name      : Cortex-A78C
model name      : Cortex-A78C
model name      : Cortex-A78C
model name      : Cortex-A78C
model name      : Cortex-X1C
model name      : Cortex-X1C
model name      : Cortex-X1C
model name      : Cortex-X1C
```

Now the FEX string is in the hypervisor functions as a leaf, so if some
utility wants the FEX version they can query that directly

Ex:
```bash
$ ./Bin/FEXInterpreter get_cpuid_fex
Maximum 4000_0001h sub-leaf: 2
We are running under FEX on host: 2
FEX version string is: 'FEX-2404-113-g820494d'
```
2024-05-01 16:27:13 -07:00
Ryan Houdek 3fda47e870 Merge pull request #3604 from alyssarosenzweig/clang-format-path
clang-format: allow overriding clang-format
2024-04-29 19:09:23 -07:00
Ryan Houdek 9c6f749ebe Merge pull request #3601 from alyssarosenzweig/jit/sbc-adc
Fix 8/16-bit ADC/SBC
2024-04-29 18:47:34 -07:00
Alyssa Rosenzweig 711ae84175 clang-format: allow overriding clang-format
it's in a weird path for me.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-29 21:43:49 -04:00
Alyssa Rosenzweig bfb06b2d55 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-29 20:00:36 -04:00
Ryan Houdek 640e911af4 Adds unit tests for ADC/SBB garbage upper data bug
Reproduces broken rendering in Final Fantasy 7 (SteamID 39140)
2024-04-29 20:00:36 -04:00
Alyssa Rosenzweig 76b5ca4bcc OpcodeDispatcher: optimize 8/16-bit adc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-29 20:00:34 -04:00
Alyssa Rosenzweig 28fa88ff39 OpcodeDispatcher: fix 8/16-bit adc/sbc flags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-29 20:00:34 -04:00
Ryan Houdek 1069cabad0 Merge pull request #3598 from Sonicadvance1/half_barrier_delete_hack
Arm64: Adds another TSO hack to disable half-barrier TSO
2024-04-26 18:24:49 -07:00
Ryan Houdek fe70ec7277 Merge pull request #3599 from alyssarosenzweig/jit/fix-faddv
JIT: fix neon vec4 faddv
2024-04-24 18:20:11 -07:00
Alyssa Rosenzweig 4a4fa64254 JIT: fix neon vec4 faddv
We were previously genrating nonsense code if the destination != source:

         faddp v2.4s, v4.4s, v4.4s
         faddp s2, v4.2s

The result of the first faddp is ignored, so the second merely calculates the
sum of the first 2 sources (not all 4 as needed).

The correct fix is to feed the first add into the second, regardless of the
final destination:

         faddp v2.4s, v4.4s, v4.4s
         faddp s2, v2.2s

Hit in an ASM test with new RA.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-24 21:13:02 -04:00
Ryan Houdek 6463054fa3 Arm64: Adds another TSO hack to disable half-barrier TSO
A feature of FEX's JIT is that when an unaligned atomic load/store
operation occurs, the instructions will be backpatched in to a barrier
plus a non-atomic memory instruction. This is the half-barrier technique
that still ensures correct visibility of loadstores in an unaligned
context.

The problem with this approach is that the dmb instructions are HEAVY,
because they effectively stop the world until all memory operations in
flight are visible. But it is a necessary evil since unaligned atomics
aren't a thing on ARM processors. FEAT_LSE only gives you unaligned
atomics inside of a 16-byte granularity, which doesn't match x86
behaviour of cacheline size (effectively always 64B).

This adds a new TSO option to disable the half-barrier on unaligned
atomic and instead only convert it to a regular loadstore instruction,
ommiting the half-barrier. This gives more insight in to how well a
CPU's LRCPC implementation is by not stalling on DMB instructions when
possible.

Originally implemented as a test to see if this makes Sonic Adventure 2
run full speed with TSO enabled (but all available TSO options disabled)
on NVIDIA Orin. Unfortunately this basically makes the code no longer
stall on dmb instructions and instead just showing how bad the LRCPC
implementation is, since the stalls show up on `ldapur` instructions
instead.

Tested Sonic Adventure 2 on X13s and it ran at 60FPS there without the
hack anyway.
2024-04-24 13:09:00 -07:00
Ryan Houdek 81a4206805 Merge pull request #3581 from neobrain/fix_libfwd_x11_libname
Library Forwarding: Fix issues with libGL's fake X11 dependency
2024-04-23 14:07:09 -07:00
Ryan Houdek a0bf6a4255 Merge pull request #3595 from alyssarosenzweig/ir/before
Factor out SetWriteCursorBefore
2024-04-23 13:34:05 -07:00
Mai 7b88b0f271 Merge pull request #3596 from Sonicadvance1/fix_fedora
Allocator: Fixes compiling on Fedora 40
2024-04-23 16:03:52 -04:00
Ryan Houdek 308488c419 Allocator: Fixes compiling on Fedora 40
This header was missing.

Either libstdc++14 or clang-18 changed includes and we were only getting
this indirectly before.
2024-04-23 12:12:33 -07:00
Ryan Houdek 81c219c212 Merge pull request #3589 from bylaws/wow-inval
WOW64 backend code invalidation fixes
2024-04-23 10:50:41 -07:00
Alyssa Rosenzweig 68e543c81d Merge pull request #3594 from Sonicadvance1/enhanced_repmovs
CPUID: Enable enhanced rep movs in more situations
2024-04-23 13:10:39 -04:00
Alyssa Rosenzweig 2372c9458b ConstProp: use SetWriteCursorBefore
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-23 13:09:45 -04:00
Alyssa Rosenzweig 1a11343f34 IREmitter: add SetWriteCursorBefore helper
This is subtle, add an ergonomic helper for it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-23 13:09:45 -04:00
Tony Wasserka cf75cdd16d Library Forwarding: Fix issues with libGL's fake X11 dependency
The library's soname is changed to libX11.so.6 and the CMake target is
renamed to libPlaceholderX11. This fixes two issues:
* Steam and mangohud can't find libX11 during startup if the library doesn't
  include a version suffix.
* Calling the CMake target libX11 overrode the true host X11 library used by
  unrelated targets (such as FEXConfig), which could cause link errors
2024-04-23 16:22:58 +02:00
Ryan Houdek 84d5b3ee59 CPUID: Enable enhanced rep movs in more situations
Instead of only enabling enhanced rep movs if software TSO is disabled,
Enable it if software tso is disabled OR memcpysettso is disabled. This
is because now we hit the fast path when memcpysettso is disabled alone
but global TSO is disabled.

Retested Hades and performance was fine in this configuration.
2024-04-21 18:50:17 -07:00
Ryan Houdek 376936c808 Merge pull request #3591 from alyssarosenzweig/ra/fix
JIT: fix ShiftFlags shuffles
2024-04-19 13:50:45 -07:00
Ryan Houdek 20bd86473b Merge pull request #3588 from Sonicadvance1/implement_smsw
OpcodeDispatcher: Implement support for SMSW
2024-04-19 07:18:45 -07:00
Alyssa Rosenzweig 932b8f38f4 JIT: fix ShiftFlags shuffles
messed up my RA.

fixes ShiftPF.asm with jit_1 with a pathological register allocation

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-18 14:11:03 -04:00
Billy Laws a7f4e99278 InvalidationTracker: Always invalidate all regions of a section on unmap
Unmapping a section will unmap the whole size initially allocated,
irrespective of how their protections are changed afterwards. Make sure
to follow this logic for invalidation too.
2024-04-18 15:16:28 +00:00
Billy Laws 7391456e48 Windows: Don't redefine existing MinGW ntdll exports 2024-04-18 15:15:11 +00:00
Billy Laws a6d061b711 InvalidationTracker: Invalidate code across all threads
When thread management was moved to the frontend, invalidation moved
from being a global operation to per-thread but the WOW64 backend wasn't
updated to account for this. Now for any invalidation event loop over
all threads tracked by the frontend and invalidate the appropriate
range.
2024-04-18 15:14:31 +00:00
Billy Laws d92580bccf WOW64: Keep track of all created threads on the frontend
This is necessary so that code can be invalidated across all threads
rather than just the initiator on any event that triggers invalidation.
2024-04-18 15:00:24 +00:00
Ryan Houdek c8704a7f71 OpcodeDispatcher: Implement support for SMSW
Found out that Far Cry uses this instruction and it is viable to use in
CPL-3. This only returns constant data but its behaviour is a little
quirky.

This instruction has a weird behaviour that the 32-bit operation does an
insert in to the 64-bit destination, which might be an Intel versus AMD
behaviour. I don't have an Intel machine available to test if that
theory is true although. This assumption would match similar behaviour
where segment registers are inserted instead of zext.

Gets the game farther but then it crashes in a `___ascii_strnicmp`
function where the arguments end up being `___ascii_strnicmp(nullptr, "Color", 5);`.
2024-04-18 07:41:39 -07:00
Ryan Houdek 5cb11aed3d Merge pull request #3587 from alyssarosenzweig/revert/waw
RCLSE: disable store-after-store optimization
2024-04-17 20:20:02 -07:00
Alyssa Rosenzweig 625d8ac177 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-17 14:54:19 -04:00
Alyssa Rosenzweig 352dcdb478 RCLSE: disable store-after-store optimization
Functional revert of 92f31648b ("RCLSE: optimize out pointless stores"), which
reportedly regressed some titles due to RA doom. We'll revisit later, leaving in
the code for when RA is ready to light this up.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-17 14:53:11 -04:00
Tony Wasserka 02ebb6e320 Merge pull request #3580 from neobrain/fix_libfwd_steamwebhelper_x11
AppConfig: Disable libGL forwarding for steamwebhelper
2024-04-17 07:41:25 +00:00
Tony Wasserka 53bed6de6a Merge pull request #3572 from pmatos/ClangFormatUtils
Clang Format file and script
2024-04-16 12:21:14 +00:00
Paulo Matos 00b81c8cbc Clang Format file and reformat target
Adds paths to ignore to clang-format, .clang-format file.
Also a wrapper script to clang-format to read .clang-format-ignore.

To format the whole tree at the root of the repository run:
`find . -iname '*.h' -o -iname '*.cpp' -exec python3 Scripts/clang-format.py -i \{\} \;`

Add reformat target to reformat the whole tree
2024-04-16 13:48:44 +02:00
Mai c126b209f1 Merge pull request #3582 from neobrain/fix_ci_x86flags
CI: Drop use of obsolete ENABLE_X86_HOST_DEBUG setting
2024-04-15 22:42:03 -04:00
Tony Wasserka 97386a7260 AppConfig: Disable libGL forwarding for steamwebhelper
This app bypasses the glX functions exported by libGL and instead sends GLX
requests directly via xcb. FEX won't support forwarding this usage pattern in
the foreseeable future.
2024-04-15 18:33:14 +02:00
Tony Wasserka a054b998c5 CI: Drop use of obsolete ENABLE_X86_HOST_DEBUG setting 2024-04-15 18:31:50 +02:00
Ryan Houdek f9097c0a92 Merge pull request #3561 from pmatos/cformat-PRCI
CI workflow to check clang-format usage on pull requests
2024-04-15 06:32:42 -07:00
Ryan Houdek 55c054bd8e Merge pull request #3578 from pmatos/Reformat2Ignore
Add second reformat to git blame ignore file
2024-04-15 06:31:13 -07:00
Paulo Matos 8799a48a0f Add second reformat to git blame ignore file 2024-04-15 15:23:49 +02:00
Paulo Matos e1efde9605 CI workflow to check clang-format usage on pull requests
Adapted from LLVM version of pr-code-format.yml.
Copies a few scripts from LLVM to External/.
Runs self-hosted on X64.

Assumes clang-format 16.0.6 for formatting.
2024-04-15 14:05:48 +02:00
Ryan Houdek 07e58f8db7 Merge pull request #3577 from pmatos/Reformat2
Reformat until fixed-point
2024-04-15 00:47:11 -07:00
Paulo Matos 905aa935f5 Reformat until fixed-point
Followup to 2b4ec88dae.
Some files needed a couple of calls to clang-format 16.0.6 to
reach a fixed point.
2024-04-15 09:40:00 +02:00
Ryan Houdek 7e97db8d98 Merge pull request #3575 from pmatos/GitBlameIgnoreRevs
Create .git-blame-ignore-revs with whole-tree reformat sha
2024-04-12 23:51:51 -07:00
Paulo Matos 340630a937 Create .git-blame-ignore-revs with whole-tree reformat sha
To enable it:
`git config blame.ignoreRevsFile .git-blame-ignore-revs`
2024-04-13 08:44:48 +02:00
Ryan Houdek 7614ac9f14 Merge pull request #3573 from pmatos/RemoveClangTidy
Remove trace of clang-tidy experiment from CMakeLists.txt
2024-04-12 17:13:53 -07:00
Ryan Houdek 1d806ac218 Merge pull request #3571 from pmatos/ReformatWholesale
Whole-tree reformat
2024-04-12 17:11:59 -07:00
Paulo Matos 2b4ec88dae Whole-tree reformat
This follows discussions from #3413.
Followup commits add clang-format file, script and blame ignore lists.
2024-04-12 16:26:02 +02:00
Tony Wasserka 028c220041 Merge pull request #3574 from pmatos/ConstPlacementLeft
Move const to the left in preparation for reformatting
2024-04-12 14:16:46 +00:00
Paulo Matos 6524716404 Move const to the left in preparation for reformatting
clang-format-16 had some issues with const placement, so we are manually changing these.
2024-04-12 16:06:57 +02:00
Paulo Matos 20559853ee Remove trace of clang-tidy experiment from CMakeLists.txt 2024-04-12 12:31:04 +02:00
Ryan Houdek a9b7ad841c Merge pull request #3570 from bylaws/ec_pt8
Enable jemalloc for ARM64EC
2024-04-11 12:57:36 -07:00
Ryan Houdek ae5e388e1a Merge pull request #3569 from bylaws/ec_pt7
FHU: Switch over win32 file operations to std::filesystem
2024-04-11 00:33:04 -07:00
Ryan Houdek 271700e9f6 Merge pull request #3568 from lioncash/const
X87: Simplify constant loading for FLD family
2024-04-11 00:32:26 -07:00
Ryan Houdek 1ba678f631 Merge pull request #3562 from Sonicadvance1/fix_rsp_store_tso
OpcodeDispatcher: Fixes disabling TSO access on RSP SIB stores
2024-04-11 00:32:14 -07:00
Billy Laws ef48700f3e FHU: Switch over win32 file operations to std::filesystem
These were broken to varying degrees across the board on win32,
just switch to their std::filesystem as windows doesn't have the
same allocator issues that require their reimplementation.
2024-04-10 22:28:12 +00:00
Ryan Houdek a0f2cae1cb Merge pull request #3567 from lioncash/veczero
IR: Remove VectorZero
2024-04-09 20:04:26 -07:00
Ryan Houdek 66cbb66732 Merge pull request #3566 from lioncash/address
OpcodeDispatcher: Add helper for making segment offset addresses
2024-04-09 17:31:43 -07:00
Billy Laws 7ff4cd9108 Update jemalloc submodule 2024-04-09 23:43:49 +00:00
Billy Laws 5ed593b59f CMake: Force enable jemalloc when targetting ARM64EC 2024-04-09 23:42:23 +00:00
Billy Laws f1f0c47f16 AllocatorHooks: Allow using jemalloc on win32 2024-04-09 23:42:23 +00:00
Lioncache 4cb2432b5c OpcodeDispatcher: Make use of new x87 constants
Now we can load these directly instead of needing to manually materialize them.
2024-04-09 10:17:15 -04:00
Lioncache 27ba66a181 IRDumper: Extend printer for NamedVectorConstant
Makes it aware of the x87 constants.
2024-04-09 10:13:35 -04:00
Lioncache 65b5281d7c IR: Add constants for FLD variants 2024-04-09 10:13:33 -04:00
Ryan Houdek 1a8b61b9fc Merge pull request #3560 from bylaws/ec-pt6
FEXCore: Support x64 -> arm64ec calls
2024-04-09 07:08:38 -07:00
Ryan Houdek f0dad86332 Merge pull request #3559 from bylaws/ec-pt5
LookupCache: Track ARM64EC page state in the code cache
2024-04-09 07:08:29 -07:00
Mai eedb120fd0 Merge pull request #3563 from Sonicadvance1/fill_spill_pairs
JIT: Adds support for spilling/Filling GPRPair
2024-04-08 23:05:11 -04:00
Lioncache 98841fe07a IR: Remove VectorZero
We have LoadNamedVectorConstant that now performs this behavior while
also being more flexible.
2024-04-08 22:42:58 -04:00
Lioncache b0aeb501f4 OpcodeDispatcher: Add helper for making segment offset addresses
There's quite a few places where the segment offset appending is open-coded
throughout the opcode dispatcher, but we can pull these out into a few
helpers to make the sites a little more compact and declarative.
2024-04-08 17:50:58 -04:00
Ryan Houdek 1616b4e77c Merge pull request #3564 from lioncash/header
DebugData: Remove header
2024-04-08 14:33:42 -07:00
Lioncache b26bf2eaf6 DebugData: Remove header
This isn't included or used anywhere, so it can be removed.
2024-04-08 16:05:40 -04:00
Ryan Houdek e91e1d5533 Merge pull request #3550 from alyssarosenzweig/ra/validate
Validate that we have no crossblock liveness
2024-04-08 12:07:23 -07:00
Ryan Houdek 574fdcef32 JIT: Adds support for spilling/Filling GPRPair
Tony noticed this last week. I encountered it this week.
Add support for spilling and filling GPR pairs.
2024-04-08 11:55:29 -07:00
Ryan Houdek 0e93fd0f3e OpcodeDispatcher: Fixes disabling TSO access on RSP SIB stores
GPR Direct/Indirect already had this and SIB version also already
supported on the load side. Fixes this missed behaviour.
2024-04-08 11:32:01 -07:00
Alyssa Rosenzweig 063954c9b3 ValueDominanceValidation: rm deadcode
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-08 13:49:35 -04:00
Alyssa Rosenzweig 7b3e031678 ValueDominanceValidation: forbid crossblock liveness
Now that we have successfully eliminated crossblock liveness from the IR we
generate, validate as much to ensure it doesn't come back. We will take
advantage of this new invariant in RA in the future.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-08 13:49:35 -04:00
Alyssa Rosenzweig a775e474d5 ValueDominanceValidation: do not validate inline constants
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-08 13:49:35 -04:00
Alyssa Rosenzweig 0e99019586 ValueDominanceValidation: actually validate
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-08 13:49:35 -04:00
Alyssa Rosenzweig d0ff3e64d9 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-08 13:49:31 -04:00
Alyssa Rosenzweig d9493e5d9b OpcodeDispatcher: fix xblock liveness in xsave/xrstr
didn't fix this hard enough before. caught by validation.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-08 13:48:29 -04:00
Alyssa Rosenzweig eb83c9e7f2 Core: use safe CondJump for self-modifying code
this ensures we put the StoreNZCV in the right block, which will fix validation
fails later in the series.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-08 13:48:29 -04:00
Ryan Houdek bb24e1419c Merge pull request #3558 from bylaws/ec-pt3
AllocatorHooks: Mark JIT code memory as EC code on ARM64EC
2024-04-08 09:32:51 -07:00
Ryan Houdek cbfa426b59 Merge pull request #3548 from alyssarosenzweig/ra/shifts-rework
Eliminate xblock liveness for shifts
2024-04-08 09:31:49 -07:00
Billy Laws 526e3e654f LookupCache: Track ARM64EC page state in the code cache
Rather than checking the actual EC bitmap in the dispatcher (~6 instrs), this
indirection through the code cache allows just 1 instr for the hot path
of calling repeated EC code/x64 code.
2024-04-08 16:08:17 +01:00
Billy Laws e03434ebde Update InstCountCI 2024-04-08 16:04:59 +01:00
Billy Laws 243bb45a68 FEXCore: Support x64 -> arm64ec calls
The frontend will provide the return logic via ExitFunctionEC, which
will be jumped to whenever there is an indirect branch/return to an addr
such that RtlIsEcCode(addr) returns true.
2024-04-06 13:20:48 +00:00
Billy Laws bd5b817c3a AllocatorHooks: Mark JIT code memory as EC code on ARM64EC
Executable mapped memory is treated as x86 code by default when
running under EC, VirtualAlloc2 needs to be used together with a
special flag to map JIT arm64 code.
2024-04-06 12:40:52 +00:00
Alyssa Rosenzweig b54d4931fb InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 95589f6172 OpcodeDispatcher: rm deferred variable shift flag calcs
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 098859caf7 OpcodeDispatcher: use _ShiftFlags for ASHR
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig c632543451 OpcodeDispatcher: use _ShiftFlags for SHRD
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 801cf72f95 OpcodeDispatcher: use _ShiftFlags for SHLD
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 650cd2c46e OpcodeDispatcher: use _ShiftFlags for SHR
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 5b48ce2228 OpcodeDispatcher: use _ShiftFlags for SHL
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 2173c26fd8 OpcodeDispatcher: add HandleShift helper
all the variable shift impls need to do this dance, make it common.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 982391ba9d IR: add ShiftFlags op
Generates flags for a variable shift as a dedicated IR op. This lets us optimize
around it (without generating control flow, relying on deferred flag infra,
etc). And it neatly solves our RA problem for shifts.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig a99c48b7a3 RedundantFlagCalculationElimination: do not eliminate if there are uses
we'll hit this with _ShiftFlags.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 90d5bd3aec unittests: add test for shift PF bug
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 59c3f96a23 unittests: add test for deferred flags + shift with cl=0
this failed on an earlier version of the series that otherwise passed ci.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 3661da1bc6 OpcodeDispatcher: calculate deferred flags before RMW on NZCV
otherwise we might have the wrong input NZCV.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 20:34:05 -04:00
Alyssa Rosenzweig 859df5e0b2 OpcodeDispatcher: optimize shl flag
This is something the new shift flag code will do. Backporting the opt since
that's stalled and this reduces the diff.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-05 19:38:44 -04:00
Ryan Houdek e8127b92e8 Docs: Update for release FEX-2404 2024-04-05 15:20:24 -07:00
Ryan Houdek 7786c23405 Merge pull request #3556 from Sonicadvance1/move_app_config
FEXCore: Fixes priority of FEX_APP_CONFIG
2024-04-05 15:18:24 -07:00
Ryan Houdek 904646e93b FEXCore: Fixes priority of FEX_APP_CONFIG
This environment variable had an incorrect priority on the configuration
system. The expectation was higher priority than most other layers.

Now the only layer that has higher priority is the environment
variables.
2024-04-05 13:10:43 -07:00
Alyssa Rosenzweig c43af8e975 Merge pull request #3553 from alyssarosenzweig/ra/shifts-rework-easy
OpcodeDispatcher: clean up shifts
2024-04-05 11:36:33 -04:00
Alyssa Rosenzweig a787daae41 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:27 -04:00
Alyssa Rosenzweig a05cc06ab4 OpcodeDispatcher: unify imm/1-bit ASHR
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig 031e756a78 OpcodeDispatcher: unify imm/1-bit SHR
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig 2a9f1ce8cb OpcodeDispatcher: unify imm/1-bit SHL
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig 8c53a9f051 OpcodeDispatcher: use LoadConstantShift for rotates
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig cf26ec7898 OpcodeDispatcher: use LoadConstantShift for SHRD
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig 582c3dae6e OpcodeDispatcher: use LoadConstantShift for SHLD
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig 2abac03ab0 OpcodeDispatcher: add LoadConstantShift helper
shows up a bunch

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig 8cc684fa12 OpcodeDispatcher: drop misinformed comment
tbnz only tests a single bit, not a mask.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig d92de1d947 OpcodeDispatcher: drop result masking for shifts
flag calcs are fine with upper garbage.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-04 07:42:15 -04:00
Alyssa Rosenzweig 202a60b77a Merge pull request #3549 from alyssarosenzweig/constprop/dce
ConstProp: drop dead code
2024-04-03 11:22:30 -04:00
Alyssa Rosenzweig aa8d04c341 Merge pull request #3551 from alyssarosenzweig/opt/negate-adds
Negate more to inline constants
2024-04-03 11:22:01 -04:00
Alyssa Rosenzweig d6425d05f3 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-02 13:57:06 -04:00
Alyssa Rosenzweig e07c81a5e7 ConstProp: also negate sub -> add
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-02 13:56:59 -04:00
Alyssa Rosenzweig fa76961873 ConstProp: negate adds -> subs
the arm ops are equiv, even though the x86 isn't (due to inverted carry).

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-02 13:53:21 -04:00
Alyssa Rosenzweig b92c206db9 ConstProp: rm your deadcode
not sure who this is supposed to be helping.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-02 13:21:48 -04:00
Alyssa Rosenzweig efff942724 ConstProp: drop my deadcode
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-02 13:21:48 -04:00
Alyssa Rosenzweig 8d32113521 ConstProp: rm relic of x86 jit
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-02 13:21:48 -04:00
Alyssa Rosenzweig 37f2b417e4 Merge pull request #3546 from alyssarosenzweig/flag/cleanup
Minor cleanups around flags
2024-04-02 11:29:19 -04:00
Alyssa Rosenzweig bd0b5eceb8 Merge pull request #3545 from alyssarosenzweig/opt/pf-scalar
Use scalar integer code to calculate PF
2024-04-02 11:28:53 -04:00
Alyssa Rosenzweig b632f7215c Merge pull request #3544 from alyssarosenzweig/ra/zero-multiple
OpcodeDispatcher: drop ZeroMultipleFlags
2024-04-02 11:27:45 -04:00
Ryan Houdek e8abc88702 Merge pull request #3542 from alyssarosenzweig/ra/rep
Eliminate xblock liveness with rep cmp/lod/scas
2024-04-02 04:24:24 -07:00
Ryan Houdek 29c6281e11 Merge pull request #3539 from alyssarosenzweig/ra/rol-ror2
rewrite ROL/ROR
2024-04-02 00:17:08 -07:00
Ryan Houdek 4214d9bda0 Merge pull request #3538 from pmatos/OffsetofOoB
Fix reference to out of bounds address in offsetof
2024-04-01 19:41:57 -07:00
Alyssa Rosenzweig 0a7b3efb41 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 16:42:50 -04:00
Alyssa Rosenzweig 067a5444dc OpcodeDispatcher: use HandleNZ00Write
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 16:42:38 -04:00
Alyssa Rosenzweig c7f159972d OpcodeDispatcher: rm pointless NZCV loads
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 16:42:38 -04:00
Alyssa Rosenzweig a70d0a5dd4 OpcodeDispatcher: rm unnecessary NZCV dirtying
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 16:42:38 -04:00
Ryan Houdek cd9ffd2045 Merge pull request #3536 from alyssarosenzweig/ra/rcl-rcr
OpcodeDispatcher: eliminate xblock liveness for rcl/rcr
2024-04-01 11:44:37 -07:00
Alyssa Rosenzweig 5c590b9a50 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 14:13:09 -04:00
Alyssa Rosenzweig eb4bb5875e OpcodeDispatcher: absorb invert into PF calculation
with xorn

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 14:12:33 -04:00
Alyssa Rosenzweig 3b052e826f OpcodeDispatcher: calculate PF with integer ops
based on clang's __builtin_parity

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 14:12:32 -04:00
Alyssa Rosenzweig 65ec191dc1 IR: add XornShift
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 14:12:32 -04:00
Alyssa Rosenzweig b1ddd8cd3b Merge pull request #3541 from alyssarosenzweig/opt/clc
optimize clc
2024-04-01 13:51:10 -04:00
Alyssa Rosenzweig f2d001e721 Merge pull request #3543 from alyssarosenzweig/ra/dead-code
RA: drop dead block interference code
2024-04-01 13:51:00 -04:00
Alyssa Rosenzweig 7852909cc4 OpcodeDispatcher: simplify IsNZCV
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 13:50:00 -04:00
Alyssa Rosenzweig fad243d3f6 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 13:48:27 -04:00
Alyssa Rosenzweig f8b68d8b5a OpcodeDispatcher: drop ZeroMultipleFlags
lot of complexity for only a single interesting case. we can massively simplify.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-04-01 13:48:11 -04:00
Ryan Houdek e2a095372e Merge pull request #3534 from Sonicadvance1/move_ir_defines
FEXCore: Move nearly all IR definitions to internal
2024-04-01 10:00:20 -07:00
Ryan Houdek 5c29c9d464 Merge pull request #3527 from Sonicadvance1/move_type_defines
Moves FHU TypeDefines to FEXCore includes
2024-04-01 08:57:22 -07:00
Ryan Houdek 3bed305660 Merge pull request #3526 from Sonicadvance1/move_codeloader
FEXCore: Moves CodeLoader to frontend
2024-04-01 07:52:02 -07:00
Ryan Houdek f6639c3594 Merge pull request #3525 from Sonicadvance1/move_cpubackend
FEXCore: Moves CPUBackend definition internal
2024-04-01 06:47:34 -07:00
Paulo Matos 96087a69fa Fix reference to OoB address in offsetof and remove rflags printout
Adjust static array size to match new size.
Remove rflags from printing code and adjust offsets - fixes
printing off-by-one error.
2024-04-01 13:13:17 +02:00
Alyssa Rosenzweig ca1ec232c9 RA: drop dead block interference code
Unused, and new RA won't use it either. Torch it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:51:11 -04:00
Alyssa Rosenzweig ad0dd34412 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:31:38 -04:00
Alyssa Rosenzweig 7b1bb159fa OpcodeDispatcher: use ForeachDirection for scas
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:31:38 -04:00
Alyssa Rosenzweig 5c7f2934de OpcodeDispatcher: use ForeachDirection for lods
eliminates xblock live

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:29:29 -04:00
Alyssa Rosenzweig 5d79d4eb50 OpcodeDispatcher: use ForeachDirection for CMPS
eliminates xblock liveness

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:29:16 -04:00
Alyssa Rosenzweig 3f66173bc7 OpcodeDispatcher: add ForeachDirection helper
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:28:56 -04:00
Alyssa Rosenzweig b64a594b16 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:01:44 -04:00
Alyssa Rosenzweig 4452f0acba ConstProp: optimize rmif with 0 for clc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 20:01:44 -04:00
Alyssa Rosenzweig 784cdd7b6b InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 19:54:14 -04:00
Alyssa Rosenzweig 1a1545da0f OpcodeDispatcher: rework rep cmp
1. pull flag calculation out of the loop body for perf
2. fully rotate the inner loop to save an instruction per iteration
3. hoist the rcx=0 jump to avoid computing df when rcx=0

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 19:54:00 -04:00
Alyssa Rosenzweig a70ea30c02 IR: add CondSubNZCV (ccmp) instruction
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 17:50:57 -04:00
Alyssa Rosenzweig 2aa1fd7fa3 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 14:47:19 -04:00
Alyssa Rosenzweig 15b86e4c5a OpcodeDispatcher: rewrite ROL/ROR
single unified implementation for ROL & ROR (instead of 4 cases). no more
deferred flags because it's easy to shoot ourselves in the foot with deferred
flags w.r.t the new RA design, and rotates are rare enough with very efficient
flag calculations such that the extra JIT overhead should be minimal to DCE the
resulting calculations later.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-31 14:47:19 -04:00
Alyssa Rosenzweig 67baff8a57 Merge pull request #3537 from Sonicadvance1/remove_vla_ra
RA: Removes VLA usage
2024-03-31 14:44:37 -04:00
Ryan Houdek fedc24be1e RA: Removes VLA usage
Just like #3508, clang-18 complains about VLA usage.

This vector is relatively small, only around 18 elements but is
semi-dynamic depending on arch and if FEXCore is targeting Linux or
Win32.
2024-03-30 16:50:04 -07:00
Alyssa Rosenzweig 2a625a467b Merge pull request #3530 from alyssarosenzweig/opt/cmpxchg-flags2
Optimize cmpxchg with flagm
2024-03-30 15:22:40 -04:00
Alyssa Rosenzweig 6f5e4fd34b OpcodeDispatcher: add non-flag calc version of ShiftVariable
more correct for rcl, etc

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 14:30:36 -04:00
Alyssa Rosenzweig bdda99e44f ConstProp: constant fold Neg
will come up with rotate in the next patch

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 14:15:11 -04:00
Alyssa Rosenzweig 9bca052146 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 14:12:59 -04:00
Alyssa Rosenzweig 706065b0e2 OpcodeDispatcher: accelerate cmpxchg with flagm
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 14:12:59 -04:00
Alyssa Rosenzweig 9fd32f07cb JIT: preserve nzcv for the slow atomic path
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 14:12:59 -04:00
Alyssa Rosenzweig deba6a1b76 JIT: add comment about unaligned backpatching
save future me some grief.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 14:12:59 -04:00
Alyssa Rosenzweig 6866c3d0ac InstCountCI: add dead cmpxchg case
not super optimizable but worth tracking.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 14:12:59 -04:00
Alyssa Rosenzweig d25ace43aa Merge pull request #3528 from alyssarosenzweig/ra/xsave-xrstor
Eliminate crossblock liveness in xsave/xrstor
2024-03-30 14:11:25 -04:00
Alyssa Rosenzweig d3b2ddf641 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:41 -04:00
Alyssa Rosenzweig 9010b3c117 OpcodeDispatcher: use neg trick for rcl smaller
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:41 -04:00
Alyssa Rosenzweig b0e001b660 OpcodeDispatcher: elim xblock live for smaller rcl
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:41 -04:00
Alyssa Rosenzweig eadacbd67b OpcodeDispatcher: elim xblock live with smaller rcr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:03 -04:00
Alyssa Rosenzweig 7de29749be OpcodeDispatcher: eliminate xblock live for rcl
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:03 -04:00
Alyssa Rosenzweig 1f3843ccad OpcodeDispatcher: eliminate xblock live for rcr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:03 -04:00
Alyssa Rosenzweig bc76df9901 OpcodeDispatcher: add non-flag calc version of ShiftVariable
more correct for rcl, etc

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:03 -04:00
Alyssa Rosenzweig 6e92cc454d ConstProp: constant fold Neg
will come up with rotate in the next patch

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-30 13:42:03 -04:00
Ryan Houdek ed3af580c5 FEXCore: Move nearly all IR definitions to internal
It has been a long time coming that FEX no longer needed to leak IR
implementation details to the frontend, this was legacy due to IR CI and
various other problems.

Now that the last bits of IR leaking has been removed, move everything
that we can internally to the implementation.
We still have a couple of minor details in the exposed IR.h to the
frontend, but these are limited to a few enums and some thunking struct
information rather than all the implementation details.

No functional change with this, just moving headers around.
2024-03-29 17:20:18 -07:00
Mai 2ad170b7d7 Merge pull request #3533 from Sonicadvance1/remove_debugstore
FEXCore: Remove DebugStore map
2024-03-29 20:19:47 -04:00
Ryan Houdek 8564290f76 FEXCore: Remove DebugStore map
This hasn't been used and is blocking refactoring more code.
2024-03-29 14:58:44 -07:00
Alyssa Rosenzweig 1d96631af7 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-29 09:58:03 -04:00
Alyssa Rosenzweig c513b9685d OpcodeDispatcher: eliminate crossblock liveness in xsave/xrstor
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-29 09:57:16 -04:00
Ryan Houdek d11a36eaea Moves FHU TypeDefines to FEXCore includes
FEXCore includes was including an FHU header which would result in
compilation failure for external projects trying to link to libFEXCore.

Moves it over to fix this, it was the only FHU usage in FEXCore/include
NFC
2024-03-29 02:54:54 -07:00
Ryan Houdek f46e88ebdb FEXCore: Moves CPUBackend definition internal
This is no longer necessary to be part of the public API. Moves the
header internally.

Needed to pass through `IsAddressInCodeBuffer` from CPUBackend through
the Context object, but otherwise no functional change.
2024-03-29 02:27:29 -07:00
Ryan Houdek 20eb338644 FEXCore: Moves CodeLoader to frontend
FEXCore no longer has a need for this since a bunch of related code was
already moved to the frontend. Move the CodeLoader now.
2024-03-29 02:24:53 -07:00
Ryan Houdek aa26b6288e Merge pull request #3522 from alyssarosenzweig/ra/cmpxchg8
OpcodeDispatcher: eliminate branch in cmpxchg pair
2024-03-27 21:56:19 -07:00
Mai 3d31291c3d Merge pull request #3510 from Sonicadvance1/fix_pthread_memleak
Linux/Threads: Fixes a stack memory leak for pthreads
2024-03-27 21:38:44 -04:00
Ryan Houdek 624bc3fce5 Merge pull request #3520 from Sonicadvance1/sleep_process
FEXLoader: Add a way to sleep a process on startup
2024-03-27 18:35:06 -07:00
Alyssa Rosenzweig d1722ab119 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-27 12:40:27 -04:00
Alyssa Rosenzweig 61758ea47d OpcodeDispatcher: eliminate branch in cmpxchg pair
In the old case:

* if we take the branch, 1 instruction
* if we don't take the branch, 3 instruction
* branch predictor fun
* 3 instructions of icache pressure

In the new case:

* unconditionally 2 instructions
* no branch predictor dependence
* 2 instructions of icache pressure

This should not be non-neglibly worse, and it simplifies things for RA.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-27 12:40:06 -04:00
Ryan Houdek 7b74ca1931 Merge pull request #3514 from alyssarosenzweig/opt/demon
rewrite Demon Addition Adjust (DAA) and other demonic opcodes
2024-03-26 23:24:00 -07:00
Ryan Houdek 24fd28ed9e Merge pull request #3511 from Sonicadvance1/more_tso_levers
FEXCore: Adds more TSO control levers
2024-03-26 23:23:41 -07:00
Ryan Houdek 970d5d5b13 Merge pull request #3509 from Sonicadvance1/allow_telemetry_redirect
Telemetry: Allow redirecting directory that data is written to
2024-03-26 23:23:05 -07:00
Ryan Houdek 79454ed8a6 Merge pull request #3507 from Sonicadvance1/fd_tracking_check
FEXLoader: Add some debug-only tracking for FEX owned FDs
2024-03-26 23:22:29 -07:00
Ryan Houdek 7f90ca53f7 Merge pull request #3505 from Sonicadvance1/telemetry_noncanonical
Telemetry: Adds tracker for non-canonical memory access crash
2024-03-26 23:21:32 -07:00
Mai 542f454630 Merge pull request #3517 from Sonicadvance1/remove_mman
FEXCore: Removes vestigial mman SMC checking
2024-03-26 23:28:05 -04:00
Ryan Houdek 4ea6305940 Merge pull request #3521 from neobrain/fix_libfwd_vkcreateinstance
Library Forwarding/vulkan: Fix query of vkCreateInstance function pointer
2024-03-26 08:59:23 -07:00
Tony Wasserka 4d8ffa2abb Library Forwarding/vulkan: Fix query of vkCreateInstance function pointer
The Vulkan specification states that querying "global commands" like
vkCreateInstance with a non-NULL instance is undefined behavior. Indeed, some
implementations will return null pointers in such cases.

Instead, we can drop the query from DoSetupWithInstance altogether, since
the library initializer will load the function pointer using dlsym instead.

Fixes #3519.
2024-03-26 16:25:51 +01:00
Ryan Houdek ade0c46845 FEXLoader: Add a way to sleep a process on startup
I find myself reimplementing this nearly monthly. Actually codify it so
I can stop reimplementing it.
2024-03-26 07:48:09 -07:00
Ryan Houdek 1450c92b60 Merge pull request #3518 from pmatos/20MFix
Put <20M in double quotes to avoid truncate error
2024-03-26 05:10:45 -07:00
Paulo Matos 53f02ee869 Put <20M in double quotes to avoid truncate error
Avoids bash assuming < is redirection.
See error in https://github.com/FEX-Emu/FEX/actions/runs/8434045431/job/23096421079#step:18:16
2024-03-26 11:47:57 +00:00
Ryan Houdek 6f29e75f67 FEXCore: Removes vestigial mman SMC checking
This wasn't actually wired up to anything ever since some refactoring
occured two years ago.
2024-03-26 02:56:26 -07:00
Ryan Houdek c1c797bcba FEXConfig: Add new TSO levers
Nice and convenient when testing applications.
2024-03-26 02:50:54 -07:00
Alyssa Rosenzweig bbc232741b InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 19:43:20 -04:00
Alyssa Rosenzweig dfe0bdd7f2 OpcodeDispatcher: rewrite DAS
exhaustively checked against the Intel pseudocode since this is tricky:

  def intel(AL, CF, AF):
      old_AL = AL
      old_CF = CF
      CF = False

      if (AL & 0x0F) > 9 or AF:
          Borrow = AL < 6
          AL = (AL - 6) & 0xff
          CF = old_CF or Borrow
          AF = True
      else:
          AF = False

      if (old_AL > 0x99) or old_CF:
          AL = (AL - 0x60) & 0xff
          CF = True

      return (AL & 0xff, CF, AF)

  def fex(AL, CF, AF):
      AF = AF | ((AL & 0xf) > 9)
      CF = CF | (AL > 0x99)
      NewCF = CF | (AF if (AL < 6) else CF)
      AL = (AL - 6) if AF else AL
      AL = (AL - 0x60) if CF else AL
      return (AL & 0xff, NewCF, AF)

  for AL in range(256):
      for CF in [False, True]:
          for AF in [False, True]:
              ref = intel(AL, CF, AF)
              test = fex(AL, CF, AF)
              print(AL, "CF" if CF else "", "AF" if AF else "", ref, test)
              assert(ref == test)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 19:43:10 -04:00
Alyssa Rosenzweig e26481e3cc OpcodeDispatcher: simplify AAM
in the area.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 19:43:10 -04:00
Alyssa Rosenzweig 86b5a2f352 OpcodeDispatcher: simplify AAD
noticed in the area.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 19:43:10 -04:00
Alyssa Rosenzweig 2bf880c43a OpcodeDispatcher: rewrite AAS
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 19:43:10 -04:00
Alyssa Rosenzweig 583d4f8f94 OpcodeDispatcher: factor out CalculateAFForDecimal
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 19:43:10 -04:00
Alyssa Rosenzweig 3ca2c4377f OpcodeDispatcher: rewrite AAA
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 19:43:10 -04:00
Ryan Houdek 32ec4a3c81 Merge pull request #3508 from Sonicadvance1/stop_using_vla
ELFParser: Stop using a VLA
2024-03-25 12:18:37 -07:00
Ryan Houdek 76983476b9 Merge pull request #3504 from Sonicadvance1/fix_loop_a16
OpcodeDispatcher: Fixes 32-bit mode LOOP RCX register usage
2024-03-25 12:18:14 -07:00
Alyssa Rosenzweig 150af80f3f Merge pull request #3512 from Sonicadvance1/panic_spilling_block
InstcountCI: Adds a block that is causing panic spilling
2024-03-25 13:13:48 -04:00
Alyssa Rosenzweig a8b59c16d6 Merge pull request #3513 from Sonicadvance1/panic_spilling_rip
RA: Adds RIP when a block panic spills
2024-03-25 13:13:15 -04:00
Alyssa Rosenzweig 949717a95f OpcodeDispatcher: rewrite DAA implementation
Based on https://www.righto.com/2023/01/

New implementation is branchless, which is theoretically easier to RA. It's also
massively simpler which is good for a demon opcode.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 13:00:59 -04:00
Alyssa Rosenzweig 693d86dd67 OpcodeDispatcher: add SetAFAndFixup helper
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 12:59:19 -04:00
Alyssa Rosenzweig ea4fce7a43 InstcountCI: add flagm primary 32-bit
track the demon opcodes

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-25 12:49:00 -04:00
Ryan Houdek 3034edb0aa RA: Adds RIP when a block panic spills
I find myself adding this every time I find a game that panic spills.
Let's just print it out.
2024-03-24 17:11:29 -07:00
Ryan Houdek c025039651 InstcountCI: Adds a block that is causing panic spilling 2024-03-24 17:10:00 -07:00
Ryan Houdek 64f47d1ec2 FEXCore: Adds more TSO control levers
Lets use control vector loadstores and memcpy/memset TSO visibility.
This just gives us a bit more configuration rather than TSO off or on.
2024-03-24 16:34:18 -07:00
Ryan Houdek ea31363221 Linux/Threads: Fixes a stack memory leak for pthreads
Same situation as the last stack leak memory fix, this is fairly tricky
since it is dealing with stack pivoting. Fixes the memory leak around
pthread stack allocations, making memory usage lower for applications
that constantly spin-up and destroy threads (Like Steam).

We need to let glibc allocate a minimum sized stack (128KB and we can't
control it) to work around a race condition with DTV/TLS regions. This
means we need to do a stack pivot once the thread starts executing.

We also need to be careful because the `PThread` object is deleted
inside of the execution thread, which was resulting in a use-after-free
bug.

There are definitely some more memory leaks that I'm still fighting, and I have
noticed in my abusive thread creation program that we might want to
change some jemalloc options to more aggressively cut down on residency.
This is just one out of many.
2024-03-24 05:22:22 -07:00
Ryan Houdek 70befc216f Telemetry: Allow redirecting directory that data is written to
This will be necessary
2024-03-24 00:47:35 -07:00
Ryan Houdek 50f62663ac ELFParser: Stop using a VLA
Clang-18 complains about this, use a vector instead.
2024-03-22 22:51:57 -07:00
Ryan Houdek 60755acef0 FEXLoader: Add some debug-only tracking for FEX owned FDs
I remember seeing some application last year where they closed a FEX
owned FD but now I don't remember what it was. This can really mess us
up so add some debug tracking so we can try and find it again.

Might be something specifically around flatpack, appimage, or chrome's
sandbox. I have some ideas about how to work around these problems if
they crop up but need to find the problem applications again.
2024-03-22 22:49:26 -07:00
Mai 002ca360f8 Merge pull request #3506 from Sonicadvance1/telemetry_rename
Telemetry: Rename old file instead of copying
2024-03-23 00:24:02 -04:00
Ryan Houdek 4952b2e16c Telemetry: Rename old file instead of copying
Since we do an immediate overwrite of the file we are copying, we can
instead do a rename. Failure on rename is fine, will either mean the
telemetry file didn't exist initially, or some other permission error so
the telemetry will get lost regardless.
2024-03-21 22:51:20 -07:00
Ryan Houdek cccf263080 InstCountCI: Update for Telemetry offset changes 2024-03-21 21:10:03 -07:00
Ryan Houdek 5a35e119fe Telemetry: Adds tracker for non-canonical memory access crash
This may be useful for tracking TSO faulting when it manages to fetch
stale data. While most TSO crashes are due to nullptr dereferences, this
can still check for the corruption case.
2024-03-21 20:47:36 -07:00
Ryan Houdek 9ab930cb26 unittests/ASM: Adds tests for loop instruction address size overrides
32-bit test would fail if the 16-bit address size override wasn't
respected.
2024-03-21 20:18:43 -07:00
Ryan Houdek 824f122680 OpcodeDispatcher: Fixes 32-bit mode LOOP RCX register usage
In 64-bit mode, the LOOP instruction's RCX register usage is 64-bit or
32-bit.
In 32-bit mode, the LOOP instruction's RCX register usage is 32-bit or
16-bit.

FEX wasn't handling the 16-bit case at all which was causing the LOOP
instruction to effectively always operate at 32-bit size. Now this is
correctly supported, and it also stops treating the operation as 64-bit.
2024-03-21 20:13:15 -07:00
Ryan Houdek 8852d94416 Merge pull request #3503 from alyssarosenzweig/opt/loop
OpcodeDispatcher: optimize LOOP/N/E
2024-03-21 20:05:50 -07:00
Mai 0c24aea27e Merge pull request #3502 from Sonicadvance1/remove_termux
Removes false termux support
2024-03-21 12:45:01 -04:00
Alyssa Rosenzweig 82ba16c6ed OpcodeDispatcher: optimize LOOP/N/E
Don't clobber NZCV.

Before/after assembly from the Primary_E1 unit test:

< 4340: [INFO] cset w20, ne
< 4340: [INFO] mrs x21, nzcv
< 4340: [INFO] cmp x5, #0x0 (0)
< 4340: [INFO] cset x22, ne
< 4340: [INFO] and x20, x22, x20
< 4340: [INFO] msr nzcv, x21
< 4340: [INFO] cbnz x20, #+0x8 (addr 0xffff896f8084)
< 4340: [INFO] b #+0x1c (addr 0xffff896f809c)
< 4340: [INFO] ldr x0, pc+8 (addr 0xffff896f808c)
---
> 4340: [INFO] csel x20, x5, xzr, ne
> 4340: [INFO] cbnz x20, #+0x8 (addr 0xfffed7308070)
> 4340: [INFO] b #+0x1c (addr 0xfffed7308088)
> 4340: [INFO] ldr x0, pc+8 (addr 0xfffed7308078)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-21 12:08:40 -04:00
Ryan Houdek 45ea0cd782 Removes false termux support
This was a funny joke that this was here, but it is fundamentally
incompatible with what we're doing. All those users are running proot
anyway because of how broken running under termux directly is.

Just remove this from here.
2024-03-20 22:04:32 -07:00
Ryan Houdek 6ce366ef35 Merge pull request #3499 from Sonicadvance1/overlapping_memcpy_unittest
unittests/ASM: Adds a test for overlapping memcpy using rep movs
2024-03-20 19:52:19 -07:00
Ryan Houdek 862d63adf2 unittests/ASM: Adds a test for overlapping memcpy using rep movs
Caused by #3478

This was missed in the review that it could cause issues. bylaws already
has a fix incoming that will get this unit test working.
2024-03-20 18:44:36 -07:00
Ryan Houdek 167896dc9d Merge pull request #3501 from bylaws/memcpy
FEXCore: Fallback to the memcpy slow path for overlaps within 32 bytes
2024-03-20 18:42:56 -07:00
Billy Laws 12fb26f9c0 Update InstCountCI 2024-03-20 21:08:59 +00:00
Billy Laws d490cb1b79 FEXCore: Fallback to the memcpy slow path for overlaps within 32 bytes
Take e.g a forward rep movsb copy from addr 0 to 1, the expected
behaviour since this is a bytewise copy is:
before: aaabbbb...
after: aaaaaaa...
but by copying in 32-byte chunks we end up with:
after: aaaabbbb...
due to the self overwrites not occuring within a single 32 bit copy.
2024-03-20 20:54:19 +00:00
Billy Laws 94fecb9dad FEXCore: Remove needless alignment checks for the mem{cpy,set} fastpath 2024-03-20 20:54:09 +00:00
Ryan Houdek 7dcacfe990 Merge pull request #3478 from bylaws/memcpy
FEXCore: Add non-atomic Memcpy and Memset IR fast paths
2024-03-18 18:56:44 -07:00
Billy Laws 29b05f6b90 Update InstCountCI 2024-03-18 23:30:19 +00:00
Billy Laws 8d4d8fe3e5 FEXCore: Add non-atomic Memcpy and Memset IR fast paths
When TSO is disabled, vector LDP/STP can be used for a two
instruction 32 byte memory copy which is significantly faster than the
current byte-by-byte copy. Performing two such copies directly after
oneanother also marginally increases copy speed for all sizes >=64.
2024-03-18 23:28:50 +00:00
Ryan Houdek ab8ee64352 Merge pull request #3497 from Sonicadvance1/movmaskb_constant
JIT: Optimize pmovmaskb with a named vector constant
2024-03-18 16:08:40 -07:00
Alyssa Rosenzweig 2a9fcc6a66 Merge pull request #3492 from Sonicadvance1/implement_prefetch
OpcodeDispatcher: Implement support for the various prefetch instructions
2024-03-18 07:49:47 -04:00
Ryan Houdek 20da1e4244 InstcountCI: Update for pmovmaskb 2024-03-17 18:52:21 -07:00
Ryan Houdek fd391b1b18 JIT: Optimize pmovmaskb with a named vector constant
I was looking at some other JIT overheads and this cropped up as some
overhead. Instead of materializing a constant using mov+movk+movk+movk,
load it from the named vector constant array.

In a micro-benchmark this improved performance by 34%.
In bytemark this improved on subbench by 0.82%
2024-03-17 18:40:46 -07:00
Mai ba3029b1f6 Merge pull request #3495 from Sonicadvance1/implement_rdpid
OpcodeDispatcher: Implement rdpid
2024-03-15 17:29:12 -04:00
Ryan Houdek 6757a80365 InstcountCI: Update for prefetch changes 2024-03-15 13:20:28 -07:00
Ryan Houdek f79991a9d8 OpcodeDispatcher: Implement rdpid
Missed this instruction when implementing rdtscp. Returns the same ID
result in a register just like rdtscp, but without the cycle counter
results. Doesn't touch any flags just like rdtscp.
2024-03-14 20:07:58 -07:00
Ryan Houdek ca6b2e43e6 Merge pull request #3491 from alyssarosenzweig/rclse/waw
RCLSE: Optimize store-after-store
2024-03-14 03:23:05 -07:00
Ryan Houdek 8a3d08e1d8 Merge pull request #3483 from neobrain/refactor_stealmemoryregion
Allocator: Cleanup StealMemoryRegions implementation
2024-03-14 03:21:09 -07:00
Ryan Houdek cd2a6ce820 Merge pull request #3469 from alyssarosenzweig/opt/df
Optimize DF representation
2024-03-14 03:18:41 -07:00
Ryan Houdek 4e269d8b80 Merge pull request #3494 from neobrain/fix_libfwd_float_as_int
Library Forwarding: Don't map float/double to fixed-size integers
2024-03-14 03:11:11 -07:00
Tony Wasserka 552e76c001 Library Forwarding: Don't map float/double to fixed-size integers
Fixes #3455.
2024-03-14 10:14:57 +01:00
Mai caff3cb799 Merge pull request #3493 from Sonicadvance1/bug_for_3478
unittests/ASM: Implements a unit test for #3478
2024-03-13 22:57:39 -04:00
Ryan Houdek 0d33dacc37 unittests/ASM: Implements a unit test for #3478
This unit test recreates the error condition that #3478 causes.
With a string operation that is a backwards copy then the optimization
will read past the end of the page and result in a crash.

Seemingly only happens with backwards string operations, but test
forward and backward in this test.
2024-03-13 18:36:19 -07:00
Ryan Houdek ba7b69eea2 InstCountCI: Adds prefetch addressing limits 2024-03-12 21:38:28 -07:00
Ryan Houdek 8056bee82b OpcodeDispatcher: Implement support for the various prefetch instructions
x86 has a few prefetch instructions.
- prefetch - One of two classic 3DNow! instructions
   - Prefetch in to L1 data cache
- prefetchw - One of two classic 3DNow! instructions
   - Implies prefetch in to L1 data cache
   - Prefetch cacheline with intent to write and exclusive ownership

- prefetchnta
   - Prefetch non-temporal data in respect to /all/ cache levels
   - Assumes inclusive caches?
- prefetch{t0,t1,t2}
   - Prefetch data with respect to each cache level
   - T0 = L1 and higher
   - T1 = L2 and higher
   - T2 = L3 and higher

**Some silly duplicates**
- prefetchwt1
   - Duplicate of prefetchw but explicitly L1 data cache
- prefetch_exclusive
   - Duplicate of prefetch

God Of War 2018 uses prefetchw as a hint for exclusive ownership of the
cacheline in some very aggressive spin-loops. Let's implement the
operations to help it along.
2024-03-12 21:37:31 -07:00
Ryan Houdek cc635a54f8 IR: Implements support for prefetch operation 2024-03-12 21:19:50 -07:00
Ryan Houdek 217d9d8c50 ARMEmitter: Fixes prfm with negative or unaligned offsets 2024-03-12 21:18:23 -07:00
Tony Wasserka a047ac1699 Allocator: Test CollectMemoryGaps instead of StealMemoryRegions and restore the original interfaces 2024-03-12 10:49:31 +01:00
Tony Wasserka bb0b114fc8 Allocator: Miscellaneous cleanups 2024-03-12 10:49:30 +01:00
Tony Wasserka ccd6c15316 Allocator: Use std::from_chars instead of parsing digits manually 2024-03-12 10:49:30 +01:00
Tony Wasserka 0a1fe1c8c2 Allocator: Parse process mappings per-line instead of per-character 2024-03-12 10:49:30 +01:00
Tony Wasserka f43fe5fd63 Allocator: Stop parsing more eagerly
This is a soft-revert of eaf83aa. That change is no longer needed, since the
stack special case is handled externally now.
2024-03-12 10:49:30 +01:00
Tony Wasserka dce9f651fd Allocator: Split off memory gap collection to a separate function
This function can be unit-tested more easily, and the stack special is more
cleanly handled as a post-collection step.

There is a minor functional change: The stack special case didn't trigger
previously if the range end was within the stack mapping. This is now fixed.
2024-03-12 10:49:30 +01:00
Tony Wasserka 0d71f169d0 Allocator: Adopt a more testable interface for StealMemoryRegions 2024-03-12 10:49:30 +01:00
Tony Wasserka 430ac0f70a Allocator: Fix format strings 2024-03-12 10:49:30 +01:00
Alyssa Rosenzweig 063b81da1d InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Alyssa Rosenzweig 7629007cfa OpcodeDispatcher: allow upper garbage on STOS
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Alyssa Rosenzweig 03c6abdad4 OpcodeDispatcher: optimize DF add
fuse the shift the right way

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Alyssa Rosenzweig c99cbe6d0a JIT: switch DF representation
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Alyssa Rosenzweig e3ee65e491 OpcodeDispatcher: use transformed DF for memset/memcpy
Use the 1/-1 representation instead of 0/1. This will be better by the end of
the series.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Alyssa Rosenzweig aee00f524c OpcodeDispatcher: use DF retrieval helpers
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Alyssa Rosenzweig a76321c6c1 OpcodeDispatcher: add DF retrieval helpers
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Alyssa Rosenzweig f7586f4459 CoreState: use x86 enums for readability
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:50:31 -04:00
Ryan Houdek e33a76a2ad Merge pull request #3489 from Sonicadvance1/linux_v6.8
Linux: Expose support for v6.8
2024-03-11 15:48:37 -07:00
Ryan Houdek c37a12e806 Merge pull request #3490 from Sonicadvance1/disable_assert
Disable assert in release
2024-03-11 15:48:18 -07:00
Alyssa Rosenzweig ed59f73a65 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:41:33 -04:00
Alyssa Rosenzweig 92f31648b9 RCLSE: optimize out pointless stores
can help a lot of x86 code because x86 is 2-address and a64 is 3-address, so x86
ends up with piles of movs that end up dead after translation

It's not a win across the board because our RA isn't aware of tied registers so
sometimes we regress moves. But it's a win on average, and the RA bits can be
improved with time.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:41:23 -04:00
Alyssa Rosenzweig 85f8ad3842 JIT: fix sha256msg1 encoding
botched move in the !tied reg case.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:41:23 -04:00
Alyssa Rosenzweig 64f71d87bb unittests: disable rdtsc test on sim
Patch written by Sonicadvance1. Unclear how this wasn't already broken, but we
need this to keep CI happy with the rest of this series.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-11 18:41:23 -04:00
Ryan Houdek ff0c7637c9 Merge pull request #3421 from pmatos/AddressingModes32
Improve 32bit ld/st addressing mode propagation
2024-03-11 15:20:20 -07:00
Ryan Houdek 54403e2146 Disable assert in release
Arguments and conditional doesn't get optimized out in release builds
for the inline function call versus the define.

Was showing up an annoying amount of time when testing.
2024-03-10 22:01:50 -07:00
Ryan Houdek d8202335e0 Linux: Expose support for v6.8
The new syscalls for futexes are the most interesting part
2024-03-10 15:48:55 -07:00
Ryan Houdek 9ec20c4bef Linux/Ioctls: Update ioctl emulation for v6.8
- v3d added an ioctl
- drm base added a new ioctl
- pvr and xe are new drivers in v6.8
2024-03-10 15:46:21 -07:00
Ryan Houdek ce7acd9b71 External/drm-headers: Update to v6.8 2024-03-10 15:45:34 -07:00
Ryan Houdek 8a607135fd Linux: Update syscalls for v6.8 2024-03-10 15:22:51 -07:00
Tony Wasserka 26a66790ab Merge pull request #3486 from neobrain/fix_libfwd_accidental_copy
Library Forwarding: Fix accidental data copying when converting from host to guest layout
2024-03-08 12:40:44 +01:00
Mai 6d94d79409 Merge pull request #3485 from Sonicadvance1/nouveau_ioctl
IoctlEmulation: Add missing nouveau ioctl
2024-03-07 07:57:47 -05:00
Tony Wasserka 2359a9899c Library Forwarding: Fix accidental data copying when converting from host to guest layout 2024-03-07 10:52:13 +01:00
Ryan Houdek aeb41e9ae2 IoctlEmulation: Add missing nouveau ioctl
The NVIF ioctl isn't publicly described in the nouveau headers and it is
required for anything to work with Nouveau.

Pass the ioctl command through without modification and hope that this
ioctl is architecture agnostic.
2024-03-05 16:05:13 -08:00
Tony Wasserka b892da72f3 Merge pull request #3484 from neobrain/feature_catch2_v3
Externals: Update Catch2 to v3.5.3
2024-03-05 19:30:30 +01:00
Paulo Matos a86f2d3e2c Improve 32bit constant usage in memory addressing
Folds reg+const memory address into addressing mode,
if the constant is within 16Kb.
Update instcountci files.
Add test 32Bit_ASM/FEX_bugs/SubAddrBug.asm
2024-03-05 14:01:32 +00:00
Tony Wasserka 6edba49784 Update Catch2 to v3.5.3 2024-03-05 12:15:29 +01:00
Ryan Houdek ee56a2cbde Docs: Update for release FEX-2403 2024-03-04 17:55:24 -08:00
Mai d66a83a98f Merge pull request #3482 from Sonicadvance1/fix_3479
Removes steamwebhelper config
2024-03-02 22:27:28 -05:00
Ryan Houdek 067b346e1b Removes steamwebhelper config
Fixes #3479

As of the Steam update on Feb 29 2024, this is no longer necessary and
actually causes steamwebhelper to abort.

Steam has started using SLR to launch steamwebhelper, which already
passes in --no-sandbox. Adding this argument twice seemingly breaks the
application since it starts complaining that it doesn't understand the
argument and closes with a SIGABRT.

With this removed Steam starts working again.
2024-03-02 13:32:08 -08:00
Ryan Houdek ea7d1697b1 Merge pull request #3480 from alyssarosenzweig/setf
Use SETF8/16 for 8/16-bit INC/DEC
2024-03-02 13:31:44 -08:00
Alyssa Rosenzweig 5fd91d34f1 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-01 19:40:53 -04:00
Alyssa Rosenzweig 11880459a5 OpcodeDispatcher: use SETF for DEC
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-01 19:40:53 -04:00
Alyssa Rosenzweig 0ef0bb2c97 OpcodeDispatcher: use SETF for INC
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-01 19:40:53 -04:00
Alyssa Rosenzweig 72edee7c6f IR: add SETF8/SETF16 ir ops
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-01 19:40:53 -04:00
Alyssa Rosenzweig cb5644cf81 InstCountCI: add multi-inst 8-bit DEC cases
would've got a bug in an earlier version of this series

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-03-01 19:38:40 -04:00
Ryan Houdek 2dd922c3ce Merge pull request #3477 from neobrain/feature_update_vulkan
Library Forwarding: Update Vulkan definitions to v1.3.278
2024-03-01 10:38:14 -08:00
Alyssa Rosenzweig b7984e8651 Merge pull request #3471 from Sonicadvance1/another_3421_bug
ASM: Another sign extend bug in #3421
2024-03-01 10:57:32 -04:00
Tony Wasserka 31e976a5bc Library Forwarding: Update Vulkan definitions to v1.3.278 2024-02-29 19:08:38 +01:00
Ryan Houdek 009ae55ff0 Merge pull request #3475 from alyssarosenzweig/opt/lock-dec
Optimize lock dec
2024-02-29 08:44:24 -08:00
Ryan Houdek 98572b9e23 Merge pull request #3473 from Sonicadvance1/remove_mov_swap
Arm64: Stop moving source in atomic swap
2024-02-29 08:44:16 -08:00
Alyssa Rosenzweig 1ab234f6dd InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-29 09:28:21 -04:00
Alyssa Rosenzweig fed5e6d546 OpcodeDispatcher: use fetchadd for atomic DEC
Avoids a NEG.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-29 09:28:21 -04:00
Alyssa Rosenzweig f27e2246e2 Merge pull request #3468 from alyssarosenzweig/opt/miscs
Misc little opts
2024-02-29 09:18:17 -04:00
Mai 4779fb74de Merge pull request #3474 from Sonicadvance1/fix_early_exit_region_reserving
Fix reserving range check
2024-02-29 07:22:07 -05:00
Ryan Houdek eaf83aa6b4 Fix reserving range check
Fixes an issue where TestHarnessRunner was managing to reserve the space
below stack again, resulting in stack growth breaking. Would typically
only show up when using the vixl simulator under gdb for some reason.

This is likely the last bandage on this code before it gets completely
rewritten to be more readable.
2024-02-29 04:02:05 -08:00
Ryan Houdek 4f8b28e83b InstcountCI: Update for swap improvement 2024-02-29 03:07:19 -08:00
Ryan Houdek c318947695 Arm64: Stop moving source in atomic swap
ldswpal doesn't overwrite the source register and only reads the bits
required for the sized operation.
Not sure exactly why we were doing a copy here.

Removing it means improving Skyrim's hottest code block, as seen in #3472
2024-02-29 03:07:05 -08:00
Ryan Houdek b3489d7262 ASM: Another sign extend bug in #3421
This time found in MGRR. It flips the problem space on its head.
2024-02-29 02:06:57 -08:00
Alyssa Rosenzweig 44d738fa93 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 10:35:34 -04:00
Alyssa Rosenzweig 811487ad98 OpcodeDispatcher: use real branch for INT
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 10:35:12 -04:00
Alyssa Rosenzweig 4f4e38ace2 OpcodeDispatcher: use real branch for rep cmps
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 10:35:12 -04:00
Alyssa Rosenzweig edd6becc56 OpcodeDispatcher: use real branch for rep scas
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 10:35:12 -04:00
Alyssa Rosenzweig e47a94cae7 OpcodeDispatcher: skip mask with shld
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 10:18:29 -04:00
Alyssa Rosenzweig cc82dba1ca OpcodeDispatcher: use mvn for AF with constants
This reduces pointless constant usage. For now, it's no net change to
instcountci, but it should make it easier to get wins later. Hopefully.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 10:18:28 -04:00
Alyssa Rosenzweig 8232669b22 OpcodeDispatcher: simplify
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 10:00:13 -04:00
Alyssa Rosenzweig 28936073c4 OpcodeDispatcher: allow upper garbage on NEG
like SUB.

due to RA silliness, this is a loss for inst count but a win for cycles.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 09:59:57 -04:00
Alyssa Rosenzweig ef2559d911 OpcodeDispatcher: allow garbage with SCAS
it's just feeding SUB flags which allow it

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-28 09:16:08 -04:00
Ryan Houdek d24446ed13 Merge pull request #3466 from Sonicadvance1/fixed_opt
OpcodeDispatcher: Don't use AddShift with no shift
2024-02-28 04:51:10 -08:00
Ryan Houdek 67f13ba927 Adds sign extending address bug that was detected when testing #3421
Doesn't quite match the libc code directly because it uses `[gs:eax]`
with both having the sign bit set and we can't deal with that with ASM
tests. So match the behaviour in a different way.
2024-02-27 19:57:22 -08:00
Ryan Houdek dc5239c003 InstCountCI: Update for previous fix 2024-02-27 19:57:06 -08:00
Ryan Houdek f346f89678 OpcodeDispatcher: Don't use AddShift with no shift
This accidentally removed optimizations elsewhere that was only checking
for Add.
2024-02-27 19:56:12 -08:00
Ryan Houdek 2f9449cb5a Merge pull request #3465 from alyssarosenzweig/icci/pa
InstCountCI: enable preserve_all
2024-02-27 16:39:46 -08:00
Ryan Houdek 139367d248 Merge pull request #3463 from Sonicadvance1/update_xxhash
Update xxhash to v0.8.2
2024-02-27 16:39:38 -08:00
Alyssa Rosenzweig fcebad51bd InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-27 12:04:29 -04:00
Alyssa Rosenzweig b50292493a InstCountCI: enable preserve_all ABI
This is what we'll actually ship (I hope), so that's the config we want to
track long-term. It's also a lot more managable resulting asm.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-27 12:03:58 -04:00
Ryan Houdek b74de53056 Merge pull request #3449 from Sonicadvance1/syscalls_passthrough_not_json
Linux: Converts passthrough syscalls to direct passthrough handlers
2024-02-27 07:50:04 -08:00
Alyssa Rosenzweig 49e798ab2b Merge pull request #3461 from alyssarosenzweig/opt/sbc
Optimize SBC
2024-02-27 11:29:45 -04:00
Ryan Houdek 151e2279af Linux: Converts passthrough syscalls to direct passthrough handlers
Reimagining of #3355 without any json generators or new concepts.

Fixes some mislabeling of system calls. Some getting inlined when they
shouldn't be, a lot not getting inlined when they can be.

This really cleans up the syscall implementation, all syscalls that can
be passthrough implementations require a very small two line
declaration.
Additionally cleans up a bit of implementation cruft where some
passthrough syscalls were using the glibc syscall handler, and some were
using the glibc implementation. We have had multiple issues in the past
where the glibc implementation does something subtly different than the
raw syscall and breaks things. Now all passthrough handlers do a system
call directly, removing at least one indirection and some ambiguity.

This makes it significantly easier to add new passthrough syscalls as
well. Only need to do a version check and add the three lines per
syscall. Which there are new syscalls incoming that we will want to add.

Tangible improvements:
- Syscalls are lower overhead than ever.
- When I'm adding more syscalls I have less chance of mucking it up.
2024-02-27 02:40:53 -08:00
Ryan Houdek 93ada89708 Linux: Move unimplement ustat and sysfs
AArch64 doesn't implement these and will return ENOSYS.
Moving them to NotImplemented so we can get a log if an application
tries to use these.
2024-02-27 02:39:36 -08:00
Mai f41674bb7d Merge pull request #3464 from Sonicadvance1/psychonauts_block
InstcountCI: Add a monster of a game block
2024-02-27 04:59:20 -05:00
Ryan Houdek 854fd70735 InstcountCI: Add a monster of a game block
Doing very little work with a bunch of instructions.
Hottest block in the Windows version of Psychonauts, it's just doing a
matrix swizzle but in the worst possible way.
2024-02-27 01:51:20 -08:00
Ryan Houdek 78a362581d Update xxhash to v0.8.2
Switches to using upstream cmake files.
2024-02-26 23:57:25 -08:00
Mai 8c0d5c6583 Merge pull request #3462 from Sonicadvance1/update_vixl4
Update vixl
2024-02-27 02:28:39 -05:00
Ryan Houdek 1c184997e7 InstcountCI: Update for vixl update 2024-02-26 23:17:52 -08:00
Ryan Houdek ccc699444d Update vixl
Removes a commit from our fork.
2024-02-26 23:16:31 -08:00
Ryan Houdek 946c805d84 Merge pull request #3459 from Sonicadvance1/fix_591
Capture a 64-bit process trying to jump to 32-bit syscall handler
2024-02-26 21:57:03 -08:00
Ryan Houdek 118b8b200e Merge pull request #3458 from Sonicadvance1/fix_635
Track unittest dependencies through to the custom target
2024-02-26 21:56:45 -08:00
Ryan Houdek aa9d7c5629 Merge pull request #3460 from Sonicadvance1/add_unittest_for_3421_bug
Adds a unittest for a bug from #3421
2024-02-26 21:56:17 -08:00
Ryan Houdek 0b34035085 Merge pull request #3439 from Sonicadvance1/allocate_first_4gb_of_64bit
FEXLoader: Allocate the second 4GB of virtual memory when executing 32-bit
2024-02-26 18:46:22 -08:00
Alyssa Rosenzweig b6bd826014 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 16:48:15 -04:00
Alyssa Rosenzweig 12cc980603 OpcodeDispatcher: shuffle adc flag order
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 16:48:07 -04:00
Alyssa Rosenzweig f3d55dd721 OpcodeDispatcher: shuffle SBC flag order
avoids clobbering nzcv

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 16:48:07 -04:00
Alyssa Rosenzweig 91cef6b76f OpcodeDispatcher: use native ADC even for 8/16-bit
we mask off the upper bits, and they agree in the lower bits.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 16:48:07 -04:00
Alyssa Rosenzweig 270cbf39b5 OpcodeDispatcher: specialize SALC
this gets rid of the awkward non-flag SBB case, which streamlines SBB. while
getting better codegen for the demon opcode (-:

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 16:33:41 -04:00
Alyssa Rosenzweig 2e0be0a5e7 OpcodeDispatcher: allow more upper garbage with adc
missed this last series.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 15:42:13 -04:00
Alyssa Rosenzweig d60c089697 OpcodeDispatcher: allow upper garbage with sbb
for the usual reasons

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 15:35:14 -04:00
Alyssa Rosenzweig e76ebeab58 OpcodeDispatcher: use 1-op "src + CF"
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 15:35:14 -04:00
Alyssa Rosenzweig 333271d490 OpcodeDispatcher: fuse sbb when flags calculated
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 15:35:14 -04:00
Alyssa Rosenzweig a750870abf OpcodeDispatcher: use fused sbcs calculations
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 15:25:03 -04:00
Alyssa Rosenzweig 15db72ef60 IR: add Sbb, SbbWithFlags ops
For fusing sbc+sbcs

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-26 15:25:03 -04:00
Ryan Houdek 9687ac51f0 Merge pull request #3424 from Sonicadvance1/safer_clone_stack_handling
Linux: More safe stack cleanup for clone
2024-02-26 06:59:27 -08:00
Alyssa Rosenzweig 5f16f357af Merge pull request #3456 from alyssarosenzweig/opt/adc
Optimize ADC
2024-02-26 09:39:37 -04:00
Ryan Houdek 4f028b8614 Capture a 64-bit process trying to jump to 32-bit syscall handler
Fixes #591

Adds a simple unittest
2024-02-26 05:37:29 -08:00
Alyssa Rosenzweig 32a4abbea7 Merge pull request #3457 from alyssarosenzweig/bug/nzcv
RedundantFlagCalculationElimination: fix missing NEG case
2024-02-26 09:31:06 -04:00
Ryan Houdek c00c9b397e Adds a unittest for a bug from #3421
When the source arguments for LoadMem/StoreMem have bit 31 set then they
are incorrectly sign extending in some instances.

Detected this when testing #3421 but I don't have a proper fix for it.
2024-02-26 00:07:19 -08:00
Ryan Houdek 6b5d8bd8c0 Track unittest dependencies through to the custom target
Fixes #635
2024-02-25 19:27:52 -08:00
Alyssa Rosenzweig 7deb4976a3 RedundantFlagCalculationElimination: fix missing NEG case
can be predicated.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-25 17:37:16 -04:00
Alyssa Rosenzweig 2cfd71c159 InstCountCI: add dead ADC test
nothing else covers this case

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-25 10:50:25 -04:00
Alyssa Rosenzweig 0f26780de0 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-25 10:50:25 -04:00
Alyssa Rosenzweig 1e153e0c81 OpcodeDispatcher: allow garbage with adcs
for the usual reasons

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-25 10:50:02 -04:00
Alyssa Rosenzweig 6994fc3a01 IR,OpcodeDispatcher,JIT: fuse adcs flags
The usual tricks, also requires introducing a bare adc op to optimize adcs to,
but we wanted that anyway!

Also support a zero source, so we can calculate "foo + CF" in one instruction to
optimize the "lock adc" cases.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-25 10:49:32 -04:00
Alyssa Rosenzweig 0ef72bf118 Merge pull request #3451 from Sonicadvance1/fix_zero_reg_regression
Fixes zero register flag generation
2024-02-24 21:08:49 -04:00
Alyssa Rosenzweig 49ca0e2181 Merge pull request #3452 from Sonicadvance1/dxvk_mgrr_hotblock
Adds MGRR hottest block on render thread
2024-02-24 21:07:25 -04:00
Ryan Houdek 947ae1c243 Adds MGRR hottest block on render thread
Was about 7% CPU time in this looping block. Has some fairly obvious
performance improvements that can be done.
2024-02-24 16:49:46 -08:00
Ryan Houdek d703f3ccee Fixes zero register flag generation
Fixes 140976d322

Adds a unit test to ensure it keeps working.
2024-02-24 16:32:25 -08:00
Ryan Houdek d8a18687e8 Merge pull request #3443 from alyssarosenzweig/opt/add-too
Fuse add + cmn -> adds
2024-02-24 15:12:16 -08:00
Alyssa Rosenzweig 045549f166 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:55:07 -04:00
Alyssa Rosenzweig 80e632db8a OpcodeDispatcher: garbage collect
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 852e3c4e93 OpcodeDispatcher: fuse XADD
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig e86547bbcb OpcodeDispatcher: fuse INC
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 883cca2e8f OpcodeDispatcher: use AddWithFlags
give it the same treatment we just gave sub.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 8540332520 IR: add AddWithFlags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig df5bdefb8a OpcodeDispatcher: merge secondary ALU with primary ALU
It's the same, stop copypasting. This gets our flag and arithmetic opts (current
and future) applied to secondary ALU too.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 3d1fb7701c OpcodeDispatcher: optimize sub
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 140976d322 OpcodeDispatcher: prep primary ALU for better flags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 9a11d3b1a2 OpcodeDispatcher: fuse NEG
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 96e652879f OpcodeDispatcher: fuse DEC
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig cc1c1dd047 OpcodeDispatcher: return result from SUB flag calculate
for fusion

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig c1d572951f OpcodeDispatcher: drop unused GenerateFlags_SUB arg
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig dd9d3264dd OpcodeDispatcher: smarten SUB flag generation
we don't need the result, we can use subs and come out ahead in practice. also a
step towards better fusion

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 2aaf957ad8 RedundantFlagCalculationElimination: DCE as we go
This is required to ensure single-iteration convergence with a sequence like:

  write C
  whatever = load C
  rmif C, whatever
  invalidate C

avoids regressing the "DEC dead" case with future work in the series.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig d459b2f9b5 IR: propagate 0 into sub
now that we have to handle it, we may as well take advantage of it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 22ab7f2b3e IR: add SubWithFlags op (arm64 subs)
with 8/16-bit handling to keep everything uniform.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig f7e32373ce JIT: allow #0 in sub
turns into neg, this will be generated via SubWithFlags -> Sub opts.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 25d422e92b JIT: use GetZeroableReg for NZCVSelect
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig cfbeece09f JIT: use GetZeroableReg for CondAddNZCV
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig a597a09825 JIT: use GetZeroableReg for SubNZCV
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 99854ff310 JIT: add GetZeroableReg helper
for inlining constant zeroes in applicable sources

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-24 15:54:49 -04:00
Alyssa Rosenzweig 44d1502b5a Merge pull request #3450 from Sonicadvance1/vector_add_loop
InstcountCI: Adds a vector addition loop from bytemark
2024-02-24 15:54:44 -04:00
Ryan Houdek c3c635e36e InstcountCI: Adds a vector addition loop from bytemark
This was interesting because it caught that we are failing loadstores
with reg+reg address generation on vectors.
2024-02-24 03:44:02 -08:00
Ryan Houdek 7e2f20cabb Merge pull request #3446 from neobrain/fix_gdb_source_map
Syscalls: Fix SourcecodeMap generation for GDB JIT integration
2024-02-24 02:46:45 -08:00
Tony Wasserka 3425b07711 Syscalls: Fix SourcecodeMap generation for GDB JIT integration
This fixes a regression from 9dd715573, which accidentally changed the
filename and set up incorrect file opening flags.
2024-02-24 11:00:23 +01:00
Ryan Houdek 9b93495d45 Merge pull request #3441 from Sonicadvance1/remove_signaldelegation_tls
Moves SignalDelegator TLS tracking to the frontend
2024-02-24 01:23:43 -08:00
Ryan Houdek cf067994f3 Merge pull request #3440 from Sonicadvance1/addressing_limits
InstcountCI: Adds addressing limitations to instcountci
2024-02-24 01:09:22 -08:00
Ryan Houdek 0a64f8a9c5 Moves SignalDelegator TLS tracking to the frontend
FEXCore doesn't need track the TLS state of the SignalDelegator, this is
a frontend concept.

Removes the tracking from the backend and keeps it in the frontend.
2024-02-24 01:07:29 -08:00
Ryan Houdek 3ac7fe3f05 Linux: More safe stack cleanup for clone
Previously: Would keep one clone thread's stack active for teardown
delaying.

With aggressive cloning and teardown, this was unsafe.
Only reap the stack when told it is safe to do so.
2024-02-24 01:05:20 -08:00
Ryan Houdek be96cb7bd0 FEXLoader: Allocate the second 4GB of virtual memory when executing 32-bit
Spurred on by #3421. To ensure that applications don't take advantage of
small address wrap around, allocate the second 4GB of virtual memory.

Some context. Linux always reserves the first 16KB of virtual address
space (unless you tinker with some settings which nobody should do).

Example of 32-bit code:
lea eax, [0xffff_0000]
mov ebx, [eax + 0x1_0000]

The address calculated by the mov will wrap around to 0x0 which will
result in SIGSEGV. If FEX messes up zero extensions then it would try to
access 0x1_0000_0000 instead.

This could result in a 32-bit application potentially accessing some FEX
memory instead of crashing.
Add this safety net which will still SIGSEGV and we will be able to see
the crash.
2024-02-24 00:54:07 -08:00
Ryan Houdek 59ec88f48d Merge pull request #3438 from Sonicadvance1/move_tls_allocation
Moves JITSymbol allocation
2024-02-23 14:49:04 -08:00
Ryan Houdek 6ec628fa31 Merge pull request #3433 from bylaws/arm64ec-pt1
Arm64Emitter: Introduce ARM64EC SRA mappings
2024-02-23 14:48:43 -08:00
Alyssa Rosenzweig 90256730c3 Merge pull request #3447 from Sonicadvance1/fix_instcountci
Fix instcountci
2024-02-22 17:24:24 -04:00
Ryan Houdek 3a0f9db512 Fix instcountci 2024-02-22 12:44:10 -08:00
Alyssa Rosenzweig 5378ae2e76 Merge pull request #3436 from alyssarosenzweig/ir/af-simplify
Simplify CalculateAF
2024-02-22 08:17:07 -04:00
Ryan Houdek cc9c80d79f Merge pull request #3445 from alyssarosenzweig/instc/fmod
InstCountCI: add FMOD block
2024-02-21 19:27:07 -08:00
Ryan Houdek 60e8da05cd Merge pull request #3442 from alyssarosenzweig/instc/witcher3
InstCountCI: add The Witcher 3 block
2024-02-21 18:56:39 -08:00
Alyssa Rosenzweig aac7fa9b58 InstCountCI: add hot scalar FMOD block
both AFP and non-AFP versions, since this is greatly affected by AFP opts.

pulled from #2563

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-21 22:36:15 -04:00
Ryan Houdek bd4a81a2a1 Moves JITSymbol allocation
This isn't actually using TLS allocations. Instead it is an allocation
tied to the InternalThreadState object.
2024-02-21 17:57:34 -08:00
Alyssa Rosenzweig a6211f29e7 InstCountCI: add The Witcher 3 block
Closes: #2688

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-21 21:44:12 -04:00
Ryan Houdek da263834f8 InstcountCI: Adds addressing limitations to instcountci
Spurred on by #3421

This does a bunch of GPR and vector loads to showcase addressing
limitations between ARM and x86.

Tests:
- 8/16/32/64-bit GPR loads
   - Both 32-bit and 64-bit addressing modes
- 32/64/128-bit Vector loads
   - Both 32-bit and 64-bit addressing modes
- Duplicate the tests for 32-bit addressing mode with a 32-bit process
   - Since it should change behaviour.

Untested:
- 8/16-bit vector loadstores since those don't exist on x86
   - 16-bit x87 integer load exists but that doesn't go through a vector
     load in FEX.
2024-02-21 16:50:00 -08:00
Ryan Houdek 3d671cba10 Allocator: Early return if past the end of the allocation range
Fixes a bug where it would eventually hit the stack region and remap the
range as RW even for ranges that don't overlap the stack.
2024-02-21 15:37:46 -08:00
Ryan Houdek d4be2dc636 Merge pull request #3434 from bylaws/arm64ec-pt3
FEXCore: Expose AbsoluteLoopTopAddress to the frontend
2024-02-21 14:31:04 -08:00
Ryan Houdek 3e5694bd06 Merge pull request #3432 from bylaws/arm64ec-pt0
CMake: Define _M_ARM_64EC when building for ARM64EC
2024-02-21 14:30:09 -08:00
Ryan Houdek 66feea9e8e Merge pull request #3422 from neobrain/feature_libfwd_wayland32
Library Forwarding: Add support for 32-bit Wayland
2024-02-21 13:41:38 -08:00
Alyssa Rosenzweig 2bcd285851 Merge pull request #3430 from Sonicadvance1/tsc_scale
Implement small TSC scaling
2024-02-21 13:16:27 -04:00
Alyssa Rosenzweig 30ff225e80 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-21 12:49:14 -04:00
Alyssa Rosenzweig 8762bc1fa3 OpcodeDispatcher: simplify CalculateAF signature
- Res is unused
- SrcSize doesn't matter since we ignore the high bits, might as well always use
  32-bit, it doesn't matter

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-21 12:48:15 -04:00
Billy Laws 5b4162b712 FEXCore: Expose AbsoluteLoopTopAddress to the frontend
ARM64EC has a shared SRA mapping between ARM64 and X64 code, so there
needs to be a public way to enter the dispatcher without refilling SRA
from the in-memory context struct.
2024-02-21 11:46:24 +00:00
Billy Laws cb5c07f4b1 Arm64Emitter: Introduce ARM64EC SRA mappings
See https://learn.microsoft.com/en-us/cpp/build/arm64ec-windows-abi-conventions?view=msvc-170
note that since mm registers are volatile there is no need to match the
mapping for them when in JIT, so they can be used as scratch regs.
Disallowed regs are also wiped on context switches, so they cannot be
taken advantage of to e.g. avoid spilling.
2024-02-21 11:18:10 +00:00
Billy Laws 3364b48f3b CMake: Define _M_ARM_64EC when building for ARM64EC 2024-02-21 11:18:07 +00:00
Tony Wasserka 67530171e6 Library Forwarding/wayland: Clean up wl_interface exchange 2024-02-21 11:44:55 +01:00
Tony Wasserka 6f00611892 Library Forwarding/wayland: Add functions required by zink + Super Meat Boy on 32-bit 2024-02-21 11:44:55 +01:00
Tony Wasserka 67941f04eb Library Forwarding/wayland: Write arguments for callbacks to guest stack 2024-02-21 11:44:55 +01:00
Tony Wasserka 42531108b7 Library Forwarding/wayland: Enable 32-bit build 2024-02-21 11:44:55 +01:00
Tony Wasserka 7edab7ee3b Library Forwarding/wayland: For 32-bit guests, support repacking wl_interface/wl_message/wl_argument 2024-02-21 11:44:55 +01:00
Tony Wasserka b9f2389d74 Library Forwarding/wayland: Avoid using global state to substitute callback tables
The pointer tracked internally by Wayland can be queried via
wl_proxy_get_listener, so we don't need our own bookkeeping.

This also changes the callback table's element type to a fixed-size uint64_t,
which makes it work for 32-bit guests.
2024-02-21 11:44:55 +01:00
Tony Wasserka 3c0b041c44 Library Forwarding/wayland: Support reading message signatures on 32-bit guests 2024-02-21 11:44:54 +01:00
Tony Wasserka 80fcd640be unittests/Library Forwarding: Test interaction between ptr_passthrough and fixed-size integer mapping 2024-02-21 11:44:54 +01:00
Tony Wasserka 0908968e87 Library Forwarding/gen: Don't register types used exclusively with ptr_passthrough annotations
This allows forwarding APIs that sparsely use non-repackable types.
2024-02-21 11:44:54 +01:00
Ryan Houdek 8e0543d9af InstcountCI 2024-02-20 12:05:44 -08:00
Ryan Houdek b902b8edab Implement small TSC scaling
Games engines are expecting >1Ghz cycle counters. Scale them to work
around the issue.

Resolves the excessive busy waiting in Unreal Engine 5 games.
2024-02-20 12:05:44 -08:00
Alyssa Rosenzweig 9c38332e7e Merge pull request #3429 from alyssarosenzweig/cleanup/nzcv-helpers
Cleanup NZCV metadata
2024-02-20 10:18:15 -04:00
Alyssa Rosenzweig 0503c89ff6 OpcodeDispatcher: use NZCV update helpers
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-19 14:12:54 -04:00
Alyssa Rosenzweig 6dd410698a OpcodeDispatcher: add helpers for updating NZCV metadata
to reduce error-prone copypaste

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-19 14:12:54 -04:00
Ryan Houdek bbac014d6d Merge pull request #3427 from Sonicadvance1/fix_vdso_crash
Fixes VDSO crash in 64-bit code
2024-02-18 18:47:41 -08:00
Ryan Houdek a723ff09c1 Fixes VDSO crash in 64-bit code
Ever since #3406 this has been crashing. Struct tail padding was saving
this before.
2024-02-17 16:43:09 -08:00
Mai 5769ffbba7 Merge pull request #3425 from Sonicadvance1/frontend_pointer
FEXCore: Add a frontend pointer to InternalThreadState
2024-02-16 19:41:41 -05:00
Mai a7c7fe4a35 Merge pull request #3426 from Sonicadvance1/thread_leak
Linux: Make sure to destroy thread object when thread shuts down
2024-02-16 19:41:23 -05:00
Ryan Houdek 80f20ad121 Linux: Make sure to destroy thread object when thread shuts down
This fixes a fairly large memor leak.
2024-02-16 15:08:29 -08:00
Ryan Houdek 808ced455d FEXCore: Add a frontend pointer to InternalThreadState
FEXCore is guaranteed to not touch this pointer and can be used by
frontends to store thread-specific data.
2024-02-15 02:06:16 -08:00
Mai 0505b30d34 Merge pull request #3423 from Sonicadvance1/fix_fd_leak
FileFormatCheck: Fixes FD leak
2024-02-14 16:01:29 -05:00
Ryan Houdek 16a5d1a6b1 FileFormatCheck: Fixes FD leak 2024-02-14 12:21:47 -08:00
Ryan Houdek 9cab746aa7 Merge pull request #3407 from neobrain/feature_libfwd_arguments_on_guest_stack
Library Forwarding: Allocate packed arguments on the guest stack if needed
2024-02-12 16:31:34 -08:00
Ryan Houdek b888bb5ce5 Merge pull request #3406 from neobrain/feature_libfwd_packed_arguments
Library Forwarding: Disable struct padding for packed arguments
2024-02-12 16:23:44 -08:00
Ryan Houdek 47218254a1 Merge pull request #3416 from alyssarosenzweig/opt/add-smol
Optimize 8/16-bit adds/subs
2024-02-12 15:52:05 -08:00
Ryan Houdek 7cb15582c2 Docs: Update for release FEX-2402 2024-02-12 10:37:55 -08:00
Ryan Houdek 930d2650f8 Merge pull request #3417 from Sonicadvance1/workaround_hang
ThreadManager: StealAndDropActiveLocks in the child forked process
2024-02-12 10:36:16 -08:00
Alyssa Rosenzweig fba5678476 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-12 12:36:48 -04:00
Alyssa Rosenzweig 68232366e4 OpcodeDispatcher: don't mask add/sub sources
not needed in the new approach

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-12 12:36:28 -04:00
Alyssa Rosenzweig d7ff1b78fb IR: handle 8/16-bit AddNZCV/SubNZCV
we can do it more effectively than the current s/w lowering.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-12 12:36:09 -04:00
Mai 780b48620b Merge pull request #3420 from Sonicadvance1/preserve_all_3419
Fix #3419
2024-02-10 23:24:38 -05:00
Ryan Houdek 4a0878fa92 Fix #3419 2024-02-10 19:55:51 -08:00
Ryan Houdek 67143ed1d1 ThreadManager: StealAndDropActiveLocks in the child forked process
The creation mutex could have been held if the parent thread was in the
middle of creating a thread when forking. This would result in a
deadlock once the fork child attempted to create another thread.

Forcefully dropping the lock in the fork child works around this
deadlock. This comes at the expense of potentially leaving resources
guarded by the thread creation mutex in an invalid state. Crashes caused
by this are easier to reason about than a delayed deadlock, though.
2024-02-10 19:51:18 -08:00
Ryan Houdek df3d6938ae Merge pull request #3410 from alyssarosenzweig/opt/nzcv-pass-2
Add NZCV+PF/AF optimization pass
2024-02-10 05:03:12 -08:00
Ryan Houdek 577372c203 Linux: Consolidate LockBeforeFork usage
Moves the CTX LockBeforeFork in to the Syscallhandler's LockBeforeFork.

This lets the syscall handler just call its own LockBeforeFork and
UnlockAfterFork functions rather than two on each call site.

Also moves the CTX->UnlockAfterFork in to the SyscallHandler's to be
consistent with the LockBeforeFork half.

No functional change.
2024-02-09 05:55:23 -08:00
Ryan Houdek d4aa64ebd1 Linux: Convert ThreadCreationMutex to forkable mutex
We will need to hold this mutex when forking.

No functional change.
2024-02-09 05:55:23 -08:00
Ryan Houdek ba41da7da0 Merge pull request #3414 from Sonicadvance1/fix_one_mutex_hang
Fixes one mutex hang
2024-02-09 05:54:40 -08:00
Alyssa Rosenzweig 806e5b8bcf Merge pull request #3415 from alyssarosenzweig/opt/testm1
Optimize test -1
2024-02-09 09:52:21 -04:00
Ryan Houdek 2480bab409 Fixes one mutex hang
When code invalidation is happening we currently have the issue that a
thread can acquire the code invalidation mutex in the middle of
invalidation. This is due to us acquiring and releasing the mutex
between each thread's code invalidation.

We need to hold the mutex for the entire duration for all thread's code
invalidation.
This fixes a rare hang on proton startup and resolves a consistent hang
on Proton application shutdown.

This now puts us on par with FEX-2312.1 with hanging.

This does not fix a relatively rare hang on fork (which also existed with FEX-2312.1).

This also does not fix the issue that the intersection of our mutexes
between frontend and backend are very convoluted. In part of the work
that is going to fix the rare fork mutex hang will change more of this.
2024-02-08 18:18:00 -08:00
Alyssa Rosenzweig de0b690672 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-08 14:11:56 -04:00
Alyssa Rosenzweig 300e2729a7 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-08 14:10:30 -04:00
Alyssa Rosenzweig ad7202e7d7 OpcodeDispatcher: optimize test -1
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-08 14:10:13 -04:00
Alyssa Rosenzweig 69466dce92 InstCountCI: add flag optimizations tests
Now that lots of instructions have optimal flag calculation in isolation, we
need to look at sequences of instructions together. These cases test various
interesting cases where concatenating the optimal instruction translations gives
something terrible for the whole block. These cases exercise the
new flag optimization pass.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 14:22:28 -04:00
Alyssa Rosenzweig 175a57dd27 OpcodeDispatcher: emit AndWithFlags directly for primary alu
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 14:22:28 -04:00
Alyssa Rosenzweig e2ce60148c OpcodeDispatcher: emit AndWithFlags directly for 2ndary alu
rely on opt pass to drop the flags.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 14:22:28 -04:00
Alyssa Rosenzweig 99660129f3 IR: implement AndWithFlags for 8/16-bit
easier to deal with in the JIT

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 14:22:28 -04:00
Alyssa Rosenzweig 308d9a751c RedundantFlagCalculationElimination: optimize rmif
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 13:06:26 -04:00
Alyssa Rosenzweig 4bd28c0ed8 RedundantFlagCalculationElimination: optimize condaddnzcv
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 13:06:26 -04:00
Alyssa Rosenzweig 8397f3ac99 RedundantFlagCalculationElimination: refine AXFLAG
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 13:06:26 -04:00
Alyssa Rosenzweig 0452bc7212 RedundantFlagCalculationElimination: optimize condjump
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 13:06:26 -04:00
Alyssa Rosenzweig 3d7ed89ffb RedundantFlagCalculationElimination: optimize NZCVSelect
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 13:06:25 -04:00
Alyssa Rosenzweig 23ab0a978e RedundantFlagCalculationElimination: also handle InvalidateFlags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 13:06:25 -04:00
Alyssa Rosenzweig 7f47a9ef0e IR: add local dead flag elimination pass
RCLSE ignores NZCV and doesn't optimize stores which doesn't help us with PF/AF
either. So, we add a new pass for dead flag elimination (cannibalizing the old
and broken dead flag elimination pass). This is a simple local optimizer that
walks each block backwards, converging in linear time & constant space in a
single iteration.

Right now, it doesn't do a ton (other than a nice reduction in silliness in
the hot Sonic block), but it provides the framework to fuse comparisons.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-06 13:06:25 -04:00
Alyssa Rosenzweig 4331753ca0 Merge pull request #3408 from alyssarosenzweig/opt/tst
Optimize TST
2024-02-06 11:28:02 -04:00
Ryan Houdek cdcc432399 Merge pull request #3411 from pmatos/AccessNullptr
Clean up access to possible nullptr
2024-02-06 04:52:28 -08:00
Paulo Matos fa8bcfd67a Clean up access to possible nullptr
Patch suggested by @Sonicadvance1
2024-02-06 12:31:38 +00:00
Mai 5e5984a29b Merge pull request #3404 from Sonicadvance1/fix_early_thread_create_tracking
Linux: Decouple thread object creation and tracking
2024-02-05 17:43:48 -05:00
Tony Wasserka 435b67aa0c Merge pull request #3409 from FEX-Emu/revert-3394-PreserveAllOption
Revert "Add cmake option DISABLE_CLANG_PRESERVE_ALL"
2024-02-05 22:40:29 +01:00
Tony Wasserka a1343e9296 Revert "Add cmake option DISABLE_CLANG_PRESERVE_ALL" 2024-02-05 22:31:45 +01:00
Alyssa Rosenzweig 2dfb7727d5 InstCountCI: add test-with-self cases
these are useful x86 instructions for e.g. branch-if-not-zero, and have
opportunities for clever translations. Add the instcountci cases to make sure we
do something reasonable for them.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-05 15:41:29 -04:00
Alyssa Rosenzweig 235f32ce8c Merge pull request #3401 from Sonicadvance1/runtime_preserve_all
HostFeatures: Supports runtime disabling of preserve_all
2024-02-05 15:34:46 -04:00
Alyssa Rosenzweig 560565aa13 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-05 15:34:11 -04:00
Alyssa Rosenzweig 2e0cb2fbd4 OpcodeDispatcher: optimize TST
it's just an AndWithFlags setting the PF.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-05 15:32:21 -04:00
Alyssa Rosenzweig 4790a7ba79 IR: add AndWithFlags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-05 15:32:21 -04:00
Mai a4a1d607d5 Merge pull request #3405 from Sonicadvance1/remove_unused_variable
SpinWaitLock: Removes unused variable in spin-loop fallback
2024-02-05 13:25:22 -05:00
Tony Wasserka df3e51fc8c Library Forwarding: Allocate packed arguments on the guest stack if needed
This is required for host-side calls to guest functions on 32-bit guests.
Since the host stack is allocated before FEX blocks memory inaccessible to
the guest, the guest would otherwise fail to read the packed argument data.
2024-02-05 18:10:34 +01:00
Tony Wasserka 06c29eab88 Library Forwarding: Disable struct padding for packed arguments
ARM64, x86 (64-bit), and x86 (32-bit) each have different alignment
requirements, so this change ensures that consistent data layout is
used for packing and unpacking.
2024-02-05 17:39:34 +01:00
Ryan Houdek 0139498072 SpinWaitLock: Removes unused variable in spin-loop fallback
Tmp was no longer being used, forgot to remove it.
2024-02-05 07:22:52 -08:00
Ryan Houdek 8cfbabde94 Linux: Decouple thread object creation and tracking
If the thread object is added to the tracking vector immediately then
there ends up being a race condition before the thread manages to fill
out the thread-specific data that only occurs at the start of the new
thread.

This manifests in a crash when a thread is allocating memory while
another thread is getting constructed. Easy fix is to defer the tracking
until the thread has setup its state.
2024-02-05 07:18:50 -08:00
Ryan Houdek 472a701e2b Merge pull request #3403 from Sonicadvance1/fix_spinlock_contended_lock
SpinLockWait: Fixes unexpected lock success
2024-02-05 06:51:42 -08:00
Mai a41ebe2d85 Merge pull request #3402 from Sonicadvance1/revert_the_revert_revert(q)revert_
Revert "Revert "FEXLoader: Moves thread management to the frontend""
2024-02-03 14:21:48 -05:00
Ryan Houdek cce6011205 SpinLockWait: Fixes unexpected lock success
With a contended unique lock, we forgot to reset the `Expected` value to
zero. This was causing a contended mutex to incorrectly succeed.

Noticed this when converting some pthread mutexes over to spinloops to
remove strace noise.

The reference wfe_mutex library I wrote didn't have this problem since
the implementation is slightly different.
2024-02-03 01:10:57 -08:00
Ryan Houdek c437129ed8 Revert "Revert "FEXLoader: Moves thread management to the frontend""
This reverts commit 5358af7794.
2024-02-03 00:57:36 -08:00
Ryan Houdek 557cb593e1 Merge pull request #3396 from alyssarosenzweig/opt/sha-stuff
Eliminate spilling in sha256rnds2 and sha1rnds4
2024-02-02 09:59:46 -08:00
Alyssa Rosenzweig afae24d870 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:55 -04:00
Alyssa Rosenzweig 8d3f0b6f02 OpcodeDispatcher: reassociate and sink W in sha1
We only need each part of W extracted in the corresponding round, so sink the
extract into the round to reduce pressure.

Further, W and E are added and then never used again. So, by reassociating we
can do the add upfront, killing W and E at the start and further reducing
pressure.

Eliminates spilling in sha1rnds4.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 60f7b9bcc4 OpcodeDispatcher: optimze sha1's 2/3 expr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig a487557173 OpcodeDispatcher: extract BitwiseAtLeastTwo
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 394b4888bb OpcodeDispatcher: reassociate and remat C0, G0
costs 2 moves and eliminates the rest of our spilling

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 142cbdd852 OpcodeDispatcher: expand, reassociate, and interleave sha256 calc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 2f9102f78d OpcodeDispatcher: expand & interleave sha256 calc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig c9824d04cb OpcodeDispatcher: sink sha256 extracts
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 9c2a569539 OpcodeDispatcher: reexpress Major in sha256
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 515aa4ce3e OpcodeDispatcher: fuse eor+ror in sha256
This reduces instructions a ton.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig f616beb992 OpcodeDispatcher: CSE sha
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 0dcf1e12b8 OpcodeDispatcher: copyprop sha logic
prepare for clever

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Alyssa Rosenzweig 2cbf544ef5 OpcodeDispatcher: expand sha logic
no functional change, just preparing for cleverness.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-02-02 13:03:07 -04:00
Ryan Houdek 0eed73beeb HostFeatures: Supports runtime disabling of preserve_all
This is used for instcountci to ensure instruction counts don't change
when a compiler supports this feature or not. Always runtime disable
when running in instcountci.

CMake option from #3394 can still be useful so leaving that in place.
2024-02-02 08:59:04 -08:00
Mai 6993f4fd8d Merge pull request #3400 from Sonicadvance1/revert_runtime_longmode_switch
Revert #3303
2024-02-02 11:53:44 -05:00
Mai 920a8db369 Merge pull request #3397 from pmatos/XCHGOp
Improve XCHG operations
2024-02-02 11:53:22 -05:00
Ryan Houdek 9c37c0f1c3 Merge pull request #3392 from neobrain/feature_libfwd_fixed_size_ints
Library Forwarding: Handle cross-architecture differences of integer types
2024-02-02 08:42:41 -08:00
Paulo Matos 4623544f69 Improve XCHG operations
Marking loads as allowing upper garbage simplifies some operations.
Update InstCountCI as well.
2024-02-02 08:16:13 +00:00
Ryan Houdek 45587278c9 InstCountCI: Update for segment changes 2024-02-01 18:17:28 -08:00
Ryan Houdek ccf1402fe6 Revert "FEXCore: Accurately store segment descriptors"
This reverts commit 8648fb1485.
2024-02-01 18:14:30 -08:00
Ryan Houdek da0e1b515a Revert "OpcodeDispatcher: Initial support for runtime long-mode switch"
This reverts commit 9e5d7aa5fe.
2024-02-01 18:14:24 -08:00
Ryan Houdek c0ec4da849 Revert "Mingw: Update for GDT"
This reverts commit ae2f98e017.
2024-02-01 18:14:19 -08:00
Mai bd1e029f35 Merge pull request #3399 from Sonicadvance1/fix_json_argumenthandler
Config: Fixes JSON parsing of "ArgumentHandler" types
2024-02-01 19:59:45 -05:00
Ryan Houdek 690cb6fa48 Config: Fixes JSON parsing of "ArgumentHandler" types
When 4d109c9ce0 fixed parsing strenum
types in the json, it also added `ArgumentHandler` types to the json
parsing. This was incorrect as those types are already stored in the
json in their decoded numerical format.

Without this change, all config options with `ArgumentHandler` will
decode as "0" which is incorrect. The main killer here is that SMCChecks
gets disabled (visible in both FEXConfig and when applications are
running) which was causing spurious failures.
2024-02-01 16:20:57 -08:00
Ryan Houdek d34302a968 Merge pull request #3398 from neobrain/fix_flt_smc_warnings
FEXLinuxTests: Fix build warnings
2024-02-01 12:52:24 -08:00
Tony Wasserka ac77376441 FEXLinuxTests: Suppress "attribute ignored" warnings for nocf_check 2024-02-01 14:43:50 +01:00
Tony Wasserka f246e9864c FEXLinuxTests: Simplify a few extern "C" declarations 2024-02-01 14:43:50 +01:00
Tony Wasserka 78cb2cd9fc FEXLinuxTests: Fix warnings about unused results 2024-02-01 14:43:50 +01:00
Ryan Houdek cec1814a09 Merge pull request #3384 from pmatos/CDQOp-Opt
Optimize CDQOp
2024-01-31 17:51:23 -08:00
Mai 4d49ac7c3d Merge pull request #3387 from alyssarosenzweig/opt/rotates
Optimize rotates
2024-01-31 18:20:40 -05:00
Alyssa Rosenzweig 6d13d9fb56 Merge pull request #3395 from pmatos/StaticAnalysis
Code cleanup - mainly dead store removal; NFC
2024-01-31 17:24:48 -04:00
Mai ae7dc250db Merge pull request #3386 from alyssarosenzweig/opt/shift
Optimize shifts a bit
2024-01-31 14:11:58 -05:00
Mai f4086b25e6 Merge pull request #3385 from alyssarosenzweig/opt/bmi
Optimize bit manipulation instructions
2024-01-31 14:07:23 -05:00
Tony Wasserka 8e1aaa0559 Library Forwarding: Avoid de-sugaring pointee types
Doing so would accidentally resolve typedefs before. This code isn't needed
now that integers are mapped to fixed-size equivalents anyway.
2024-01-31 19:16:19 +01:00
Tony Wasserka a4c1aaa6bc Library Forwarding: Fix build for clang < 16 2024-01-31 19:14:57 +01:00
Ryan Houdek 37a611bcd6 Merge pull request #3394 from pmatos/PreserveAllOption
Add cmake option DISABLE_CLANG_PRESERVE_ALL
2024-01-31 00:58:30 -08:00
Paulo Matos e4560ed0c8 Code cleanup - mainly dead store removal; NFC
scan-build found a few dead stores that can be easily cleaned-up
2024-01-31 08:35:55 +00:00
Paulo Matos 6d58ea31b9 Add cmake option DISABLE_CLANG_PRESERVE_ALL
Forces disabling use of __attribute__((preserve_all)).
Until CI uses clang17, where this attribute was added, instcountci fails
when FEX is compiled with clang>=17.
2024-01-31 08:29:20 +00:00
Ryan Houdek 3db31a655c Merge pull request #3393 from Sonicadvance1/fix_relative_data
InstCountCI: Fixes test to not use relative data
2024-01-30 23:43:26 -08:00
Alyssa Rosenzweig e9a6bfc077 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig 5a143c016c unittests: add ASM test for blsmsk flags
failing on main

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig fd1307eea3 unittests: add BLSI flag test
passing on main, but kinda by accident?

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig d9f08d4bdf unittests: add BLSR flag unit test
failng on main

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig f3eee8f305 OpcodeDispatcher: optimize bextr's length sanitize
reordering the operations saves an immediate move.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig f66085f4a7 OpcodeDispatcher: optimize bextr's (1 << x) - 1
little algebraic trick I cribbed from llvm

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig c9461d9997 OpcodeDispatcher: optimize BEXTR flag setting
use native test.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig d5eb99fac8 OpcodeDispatcher: optimize popcount flags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig f3175848b1 OpcodeDispatcher: use lzcnt flag gen for tzcnt
as far as flags go, they're identical: set ZF for zero output, set CF for output
= DestSize, undef the rest. merge the impls, so we get the optimized lzcnt impl
for tzcnt.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig 3dd597a591 OpcodeDispatcher: optimize lzcnt
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig e8e05252f0 OpcodeDispatcher: optimize BLSI
and explain why the suss thing we did before was actually right all along.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig 93cef53ec0 OpcodeDispatcher: optimize blsr flags
reorder to avoid nzcv clobber

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig 3a19133267 OpcodeDispatcher: fix inverted BLSR carry
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig 9b309b2102 OpcodeDispatcher: optimize blsmsk flags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig fe88b904c9 OpcodeDispatcher: fix missing SF set with blsmsk 2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig 2e63c6d547 OpcodeDispatcher: fix inverted CF with blsmsk
CF set if SRC = 0

per https://www.felixcloutier.com/x86/blsmsk

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:28:06 -04:00
Alyssa Rosenzweig 4cc59ba8f5 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:26:59 -04:00
Alyssa Rosenzweig 0bc9e1a409 OpcodeDispatcher: clobber OF with shift immediate
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:26:59 -04:00
Alyssa Rosenzweig 338f12845d OpcodeDispatcher: save a constant in shld
one weird trick

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:26:59 -04:00
Alyssa Rosenzweig b3ae81f75f OpcodeDispatcher: allow garbage on shld shift
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:26:59 -04:00
Alyssa Rosenzweig c1a1c37980 OpcodeDispatcher: mark ideas to improve SHLD
a bit tricky right now.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:26:59 -04:00
Alyssa Rosenzweig 56dd9f23d1 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig fb6f850bb4 OpcodeDispatcher: remove rcl sub
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig b6d8749525 OpcodeDispatcher: remove select from rcl
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig d3f1397325 OpcodeDispatcher: eliminate constants in RCR
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 0a164428fa OpcodeDispatcher: eliminate select in RCR
the nzcv clobber I actually came ofr

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 7496175100 OpcodeDispatcher: optimize 32-bit rcl/rcr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 0616a9cef1 OpcodeDispatcher: eliminate move in rcr 1-bit
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 97f8775354 OpcodeDispatcher: optimize <32-bit rcr op1
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig c92099aa98 OpcodeDispatcher: fuse orlshl in rcr 1-bit
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 7c288b09f1 OpcodeDispatcher: rmif mask rcl smaller OF
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 680af7b1b0 OpcodeDispatcher: rcr op 8x1 cleanup
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 349bc9efab OpcodeDispatcher: unify rcr op 1bit codepaths
get additional opt for <32-bit

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig ad5c3cb268 OpcodeDispatcher: rmif mask for OF in rcr smaller
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig be8d37ef3d OpcodeDispatcher: optimize 32-bit rol/ror imm
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 6ad2514bfe OpcodeDispatcher: rmif mask rcl smaller cf
better on flagm. extra moves on non-flagm but, meh.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 3fa6129a14 OpcodeDispatcher: rmif mask rcr smaller cf
and do some constant folding to do so more.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig a57cebaf58 OpcodeDispatcher: skip OF calc for constant rotate >= 2
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 34fdb14da1 OpcodeDispatcher: add and use AndConst
this skips the constant folding, which saves the branching in the rotate
immediate implementations.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 974baca09c OpcodeDispatcher: allow upper garbage with rcl/rcr smaller
we're masking immediately to something smaller

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig f22094a493 OpcodeDispatcher: use a branch for 8/16-bit rotate flags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig d979b3a1da OpcodeDispatcher: note idea to further optimize rcl
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Alyssa Rosenzweig 6d82c957fa OpcodeDispatcher: fuse orlshl in rcl
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-30 22:22:57 -04:00
Ryan Houdek 28715a1df6 InstCountCI: Fixes test to not use relative data
JIT code and loaded code memory offsets aren't consistent run to run
which causes code generation consistency issues.
Instead use a register plus an immediate that can't fit in a single
16-bit encoding to get similar register allocation to RIP relative
addresses.

At some point it will probably be nice to support consistent placement
of both guest and host code so RIP relative can be used, but we don't have
support in our backend to do that.
2024-01-30 17:40:36 -08:00
Tony Wasserka b40d8f5679 unittests/Library Forwarding: Add tests for fixed-size integer mapping 2024-01-30 19:54:45 +01:00
Ryan Houdek 2884337d85 Merge pull request #3391 from neobrain/fix_libfwd_custom_repack_nonstruct
Library Forwarding: Don't attempt custom repacking for non-struct types
2024-01-30 08:52:21 -08:00
Ryan Houdek 3036f3b3ff Merge pull request #3390 from pmatos/TestHarnessArgs
Check that path arguments to TestHarnessRunner exist
2024-01-30 08:52:14 -08:00
Paulo Matos b02ed40b5a Check that path arguments to TestHarnessRunner exist 2024-01-30 16:37:31 +00:00
Tony Wasserka 0114c874a7 unittests/Library Forwarding: Respect silent flag when compiling SourceWithAST 2024-01-30 17:22:23 +01:00
Tony Wasserka 3ff3dc8769 Library Forwarding: Add unit test for diverging parameter type sizes 2024-01-30 17:22:23 +01:00
Tony Wasserka d04c94fe80 Library Forwarding/gen: Map integers to fixed-size equivalents on guest 2024-01-30 17:22:23 +01:00
Tony Wasserka 0ebf260ed0 Library Forwarding: Consider struct metadata equal if it only differs in integer member type names 2024-01-30 17:22:23 +01:00
Tony Wasserka 28ae84cf60 Library Forwarding: Allow implicit conversions between various integer types
Generally, implicit integer conversions are prohibited for data wrapped in
guest_layout/host_layout, but a few types are exceptional:
* char vs signed char vs unsigned char vs other 8-bit ints
* wchar_t vs other 32-bit ints
* size_t vs uint32_t (32-bit only)
* long long vs other 64-bit ints
* long vs long long (64-bit only)

These combinations have the same data size, so conversions between them are
explicitly allowed now.
2024-01-30 17:22:23 +01:00
Tony Wasserka 26007168b0 Library Forwarding: Extend function pointer interface to take separate guest parameter lists
Some types (notably size_t on 32-bit) have different sizes on the guest than on
the host. This template function must be aware of these differences, so a
second parameter list with fixed-size types must be provided to describe the
guest types.

Note that this information can't be queried through type traits: To a C++
compiler, size_t is indistuingishable from uint64_t. For this reason, the
correct guest type must indeed be provided externally.
2024-01-30 17:22:23 +01:00
Tony Wasserka a48237e218 Library Forwarding: Don't attempt custom repacking for non-struct types 2024-01-30 17:18:54 +01:00
Mai fa3352004e Merge pull request #3381 from alyssarosenzweig/opt/masking
Allow upper garbage on a bunch of instructions
2024-01-30 10:07:53 -05:00
Mai b9378856da Merge pull request #3389 from Sonicadvance1/panic_spill_libnss3
instcountci: Adds panicspill test from steamwebhelper
2024-01-30 10:05:31 -05:00
Paulo Matos 515e98f4cb Update instruction counts 2024-01-30 10:47:56 +00:00
Ryan Houdek ce2924731e vixl/simulator: Enlarge simulator stack size
Simulator stack size defaults to 8KB. This new unit test requires at
least 15360 stack size. Just push it up to 8MB.
2024-01-29 19:48:38 -08:00
Ryan Houdek c099fd01f1 instcountci: Adds panicspill test from steamwebhelper
This block of code causes our RA to panic spill and it gets hit my
steamwebhelper.
We should be able to spill without panic on this code at some point.
2024-01-29 19:48:38 -08:00
Ryan Houdek 36250b10f6 InstCountCI: Sanitize out adrp and adr
First time usage of adrp and adr, need to sanitize it.
2024-01-29 19:32:04 -08:00
Ryan Houdek bc67910ee4 Merge pull request #3382 from pmatos/TypoFix
Fix typos; NFC
2024-01-29 16:10:14 -08:00
Ryan Houdek 79526b9c9e Merge pull request #3379 from neobrain/fix_fexconfig_paths
FEXConfig: Initialize paths before trying to read configuration files
2024-01-29 16:01:12 -08:00
Mai 31a4158957 Merge pull request #3383 from alyssarosenzweig/opt/ptest
Optimize PTEST and VTESTP
2024-01-29 13:30:53 -05:00
Mai 58f3d3caf5 Merge pull request #3380 from alyssarosenzweig/opt/pdep
Optimize PDEP
2024-01-29 13:27:15 -05:00
Alyssa Rosenzweig be52676b3b InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:24:11 -04:00
Alyssa Rosenzweig ae48228943 OpcodeDispatcher: optimize vtestps/vtestpd
I don't really care about AVX but do the same thing we did for vptest.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:24:11 -04:00
Alyssa Rosenzweig e8e35e48c7 OpcodeDispatcher: optimize ptest with tst
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:20:17 -04:00
Alyssa Rosenzweig 8b8f27a88f OpcodeDispatcher: optimize ptest with umaxv
to check if the vector is zero, umaxv its elements and check if the reduced
scalar is zero.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:20:17 -04:00
Alyssa Rosenzweig 8e7906a665 IR: add UMaxV
will be used to accelerate ptest

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:19:22 -04:00
Paulo Matos 85105a6af3 Update instcountci json file 2024-01-29 17:18:02 +00:00
Paulo Matos 027fbbf051 Optimize CDQOp 2024-01-29 17:18:02 +00:00
Paulo Matos ca31a0404c ConstProp should generate 32bit constants when required 2024-01-29 17:15:47 +00:00
Paulo Matos f644959c7c Fixing some typos; NFC 2024-01-29 17:14:53 +00:00
Alyssa Rosenzweig 10cca02656 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:13:44 -04:00
Alyssa Rosenzweig 16a54742e6 OpcodeDispatcher: optimize 32-bit tzcnt
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig ad9aa0bc87 OpcodeDispatcher: optimize 32-bit lzcnt
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig bd2b3f35a3 OpcodeDispatcher: optimize 32-bit popcnt
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig 50169ce640 OpcodeDispatcher: optimize 32-bit pext
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig baae2d68f9 OpcodeDispatcher: optimize 32-bit bextr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig ad8d038b8a OpcodeDispatcher: optimize 32-bit blsi
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig 820932e3c7 OpcodeDispatcher: optimize 32-bit blsmsk
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig 6f11f2e6f4 OpcodeDispatcher: optimize 32-bit blsr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:11:25 -04:00
Alyssa Rosenzweig 409b6ff6ef InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:08:30 -04:00
Alyssa Rosenzweig f5ad7682c3 OpcodeDispatcher: optimize 32-bit pdep
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:06:56 -04:00
Alyssa Rosenzweig 04805f351b JIT: rewrite pdep implementation
- use better algorithm that is O(# set bits) instead of O(# total bits)
- eliminate spilling by careful management of our temporaries
- fix nzcv clobber bug (whoops)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-29 13:06:56 -04:00
Ryan Houdek 8e3d4a3e02 Merge pull request #3375 from Sonicadvance1/implement_efault_support
Linux: Implements a fault safe memcpy routine
2024-01-26 02:15:42 -08:00
Ryan Houdek 0913741343 Linux: Disable EFAULT handler until we find something that uses it. 2024-01-26 01:54:09 -08:00
Ryan Houdek 929193c16c Linux: Implements support for EFAULT with ppoll's timeout
Only need to handle the timeout structure, the rest is handled in the
kernel itself.
2024-01-26 01:54:09 -08:00
Ryan Houdek 69ff984a82 Implements a FEXLinuxTest for testing ppoll EFAULT behaviour 2024-01-26 01:54:09 -08:00
Ryan Houdek c5ffc0664d Linux: Implements a fault safe memcpy routine
We are required in our syscall emulation to handle cases where pointers
are invalid. This means we need to pessimistically assume a memcpy will
fault when reading application memory.

This implements a signal handler based approach to catching the SIGSEGV
on memcpy and returning an EFAULT if it faults.
2024-01-25 13:50:55 -08:00
Tony Wasserka c1ef11f034 FEXConfig: Initialize paths before trying to read configuration files 2024-01-25 15:54:25 +01:00
Mai 750b0b70bc Merge pull request #3356 from Sonicadvance1/modify_code_lock
Jitarm64: Implements spin-loop futex for JIT blocks
2024-01-23 13:46:59 -05:00
Ryan Houdek 56d8080ec9 Merge pull request #3345 from Sonicadvance1/fix_syscall_registers
OpcodeDispatcher: Fixes syscall rcx/r11 generation
2024-01-22 15:21:13 -08:00
Ryan Houdek c0be974272 Merge pull request #3368 from bylaws/preprcr
FEXCore: Fix RCL/RCR shift wraparound behaviour
2024-01-21 13:44:49 -08:00
Ryan Houdek 0e97f8fb2b Merge pull request #3366 from bylaws/prep222
FEXCore: Use TMP1-4 for values that need preserving across spills
2024-01-21 13:19:34 -08:00
Billy Laws c4c10d0a16 instcountci: update 2024-01-21 18:15:50 +00:00
Billy Laws 576cf61a96 unittests: Test RC{R,L} wraparound behaviour 2024-01-21 18:15:50 +00:00
Billy Laws e323938173 FEXCore: Fix RCL/RCR shift wraparound behaviour
This ends up being cleaner to handle outside of
CalculateFlags_ShiftVariable as constant masking is only needed for
RCL/RCR.
2024-01-21 18:15:50 +00:00
Billy Laws 407e26bfee FEXCore: Use TMP1-4 for values that need preserving across spills
The ARM64EC SRA layout will use x0-3 for x86_64 registers, as such any
arguments passed to C ABI functions need to proxy their arguments
through the temporaries and move as appropriate.
2024-01-21 16:21:13 +00:00
Ryan Houdek d04f3a288e Merge pull request #3377 from neobrain/feature_thunk_flip_nullcheck_order
Library Forwarding: Flip order of nullptr check and custom repacking
2024-01-20 06:47:53 -08:00
Ryan Houdek 3bfb4bedd5 Merge pull request #3374 from neobrain/feature_thunk_test_repack_vs_assume_compatible
Library Forwarding: Test interaction of struct repacking and assume_compatible_data_layout
2024-01-20 00:15:51 -08:00
Tony Wasserka fb15cf0db4 Thunks: Flip order of nullptr check and custom repacking 2024-01-19 11:25:07 +01:00
Tony Wasserka b08d372c78 Thunks: Fix definition of GUEST_THUNK_LIBRARY
The -deps target is the wrong target to add this to, since its compile flags
are propagated to both Guest.cpp and Host.cpp. Instead, define the flag only
when processing files within a guest context.
2024-01-19 11:17:44 +01:00
Tony Wasserka acf48e29b2 FEXLinuxTests/Thunks: Test interaction of struct repacking and assume_compatible_data_layout 2024-01-19 11:17:44 +01:00
Ryan Houdek 162733130c Merge pull request #3376 from Guztaver/patch-1
Improvements to the Dockerfile
2024-01-19 01:53:49 -08:00
Gustavo Muniz f6ff6c8b44 Improvements to the Dockerfile
I added the `pkg-config` that is missing in the Dockerfile, and make the git clone the repository direct from the image, so the user don't need to clone it manually and after that pulls inside the Dockerfile!
2024-01-19 02:27:03 -03:00
Ryan Houdek 6efc4a967c Merge pull request #3373 from neobrain/feature_thunk_custom_repacking
Library Forwarding: Implement assisted struct repacking
2024-01-18 13:56:20 -08:00
Ryan Houdek a6c57f71e9 SpinWaitLock: Fixes potential extra wait that would occur on contended lock
We had a chance of doing an additional bogus wfe if the expected value
was hit in one iteration of a loop. Not the biggest problem on current
hardware where WFE only ever sleeps for 1-4 system cycles, but on future
hardware where WFE might actually sleep for longer then this could have
been an issue.
2024-01-17 10:41:16 -08:00
Ryan Houdek 2af7e997f4 Spinlocks: Fix assembly
Need to have a source be +r so it doesn't get overwritten.
2024-01-17 10:19:38 -08:00
Ryan Houdek ab6c00bbcf FEXCore/Utils: Rename FutexSpinWait to SpinWaitLock 2024-01-17 10:19:38 -08:00
Ryan Houdek e18453cb57 Jitarm64: Implements spin-loop futex for JIT blocks
This will ensure that multiple concurrent SIGBUS handlers in the same
code block doesn't modify the same code.
2024-01-17 10:19:38 -08:00
Ryan Houdek 39f49782da Arm64: Move ParanoidTSO checks up out of the non-paranoid code bath 2024-01-17 10:19:38 -08:00
Ryan Houdek 2c5dd20f3c FutexSpinWait: Implement spin-loop Unique mutex. 2024-01-17 10:19:38 -08:00
Ryan Houdek 136fa78825 FEXCore: Implements an efficient spin-loop API
This will only be used internally inside of FEXCore for efficient shared
codecach backpatch spin-loops.
2024-01-17 10:19:38 -08:00
Tony Wasserka 190d8020ff FEXLinuxTests/Thunks: Add tests for assisted struct repacking 2024-01-15 20:40:13 +01:00
Tony Wasserka 6eaeb48fac Thunks/gen: Implement assisted struct repacking
This can be used to allow automatically handling structures that require
special behavior for one member but are automatically repackable otherwise.
The feature is enabled using the new custom_repack annotation and requires
additional repacking functions to be defined in the host file for each
customized member.
2024-01-15 20:40:13 +01:00
Ryan Houdek f956f008ea Merge pull request #3372 from alyssarosenzweig/opt/cmpxchg-review
Optimize GPR cmpxchg
2024-01-15 05:11:12 -08:00
Ryan Houdek be4d1a8860 Merge pull request #3365 from Sonicadvance1/pidof
Tools: Adds new FEXpidof tool
2024-01-15 05:10:56 -08:00
Ryan Houdek 8ff4b525fc Merge pull request #3367 from bylaws/prepwow
Windows: Commonise WOW64 logic that can be shared with ARM64EC
2024-01-15 05:09:50 -08:00
Tony Wasserka 832072362b Merge pull request #3371 from neobrain/feature_thunk_automatic_repacking
Library Forwarding: Implement automatic argument repacking
2024-01-15 11:23:42 +01:00
Ryan Houdek 1f7a619c79 OpcodeDispatcher: Fixes syscall rcx/r11 generation
Noticed this while writing #3342.

Fixes #3343

The syscall instruction is defined in the documentation that it will set
RCX to the next instruction's RIP and R11 to be RFLAGS. We entirely
skipped this which I noticed while writing unit tests.

Adds unittests to test both 32-bit and 64-bit behaviour because our
helper shares code with both.

I don't know if anything actually relied on this behaviour but we should
definitely support it.
2024-01-12 19:14:30 -08:00
Ryan Houdek ddfd393789 Tools: Adds new FEXpidof tool
This behaves exactly like pidof but only searches for FEX applications.
This fixes a long standing annoyance of mine that pidof doesn't work for
FEX. This behaves exactly like pidof but knows how to decode the command
line options to pull out the program data.

If the Linux kernel ever accepts the patches for binfmt_misc to change
how the interpreter is handled then this will become redundant, but
until that happens here is a utility that I want.
2024-01-12 19:05:37 -08:00
Alyssa Rosenzweig d9985eb877 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-12 12:23:34 -04:00
Alyssa Rosenzweig 58127bd0e8 OpcodeDispatcher: optimize trivial cmpxchgs
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-12 12:23:34 -04:00
Alyssa Rosenzweig e8945dfb6d OpcodeDispatcher: optimize gpr cmpxchg
NZCV stuff.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-12 12:03:28 -04:00
Alyssa Rosenzweig 0a6e875329 InstCountCI: add nontrivial gpr cmpxchg cases
cmpxchg handles rax specially, so cmpxchg with dest=rax is a special case. test
also the general case.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-01-12 11:50:43 -04:00
Tony Wasserka cd4578de27 FEXLinuxTests/thunks: Add tests for automatic repacking 2024-01-12 14:47:10 +01:00
Tony Wasserka 295fe03eef unittests/ThunkLibs: Add simple struct repacking tests
These just ensure that two specific but common patterns continue to compile.
2024-01-12 14:47:10 +01:00
Tony Wasserka 4cbaad298a Thunks/gen: Repack out-parameters on exit
To avoid performance traps, several conditions must hold for exit repacking
to apply:
* Argument must be a pointer
* The pointee type must have differing data layout between guest and host
* The pointee type must be non-const

Arguments that don't meet the first two conditions are safe *not* to repack
on exit, since they're either passed by copy or have consistent data layout.
The third condition is a heuristic: In principle, an API function could modify
data through nested pointers even if the argument pointer is const. However,
automatic repacking is not supported for such types anyway, so this is a safe
heuristic to use.
2024-01-12 14:47:10 +01:00
Tony Wasserka dc477c3bd7 Thunks/gen: Implement automatic struct (entry-)repacking 2024-01-12 14:47:10 +01:00
Tony Wasserka 52419d9911 Thunks/gen: Emit metadata to check if a type has consistent data layout 2024-01-12 14:47:10 +01:00
Ryan Houdek 8c3163096b Merge pull request #3363 from Sonicadvance1/fix_label_allocations
ArmEmitter: Support single use forward labels
2024-01-12 00:26:31 -08:00
Billy Laws 9f311cd97e CommonWindows: Split out code invalidation logic from WOW64
This will also be used by FEX ARM64EC module.
2024-01-12 00:12:26 +00:00
Billy Laws 1115ce4a95 CommonWindows: Introduce, and split out CPU feature logic from WOW64
This can also be used by the FEX ARM64EC module.
2024-01-12 00:12:26 +00:00
Billy Laws 03b802cf8e Windows: Use MinGW importlib functions when possible for ntdll
Switch to only defining imports that are not part of MinGW in the FEX importlib
definitions, this is necessary to avoid linker problems with ARM64EC.
2024-01-12 00:12:26 +00:00
Ryan Houdek 615cfe0246 Merge pull request #3361 from Sonicadvance1/decompose_std_function
FEXCore: Decompose some std::function usage to regular pointers
2024-01-10 16:55:29 -08:00
Ryan Houdek dae16aaf18 Merge pull request #3362 from Sonicadvance1/glibc_alloc
Fixes some new glibc allocations that cropped up
2024-01-10 02:05:24 -08:00
Ryan Houdek 3d5f876585 Fixes some new glibc allocations that cropped up
I guess this was handled by brk things before.
2024-01-09 13:55:04 -08:00
Ryan Houdek 37102400b5 Arm64: Switches uses of forward label over to SingleUse if possible
Primary goal for this is to ensure that the delinker doesn't need to
allocate any memory. This delinker can end up getting hit heavily with
JIT code so we don't want it to be allocating memory.
2024-01-08 22:18:20 -08:00
Ryan Houdek c01e6283ae CodeEmitter: Support a single use forward label
Currently all uses of the forward label calls in to jemalloc to allocate
memory. This allows a forward label that doesn't require any memory
allocation, which is the common case in FEX.
2024-01-08 22:18:20 -08:00
Ryan Houdek 248dc97993 FEXCore: Decompose some std::function usage to regular pointers
The delinker step of the JIT was using std::function with capture
lambdas that required memory allocation when unnecessary.
Because the compiler can't see through our std::function usage it could
never decompose these by itself.

By passing the Thread's frame and record to the function as arguments
then we can have the signature be a raw function pointer.

This fixes an area of concern from:
https://github.com/FEX-Emu/FEX/blob/main/docs/ProgrammingConcerns.md#stdfunction-and-lambdas
2024-01-06 19:39:54 -08:00
Ryan Houdek d488592eda Merge pull request #3339 from Sonicadvance1/pass_thread_unaligned_fault_handler
FEXCore: Pass thread object to HandleUnalignedAccess
2024-01-04 18:20:37 -08:00
Ryan Houdek 743df8dfae Merge pull request #3327 from Sonicadvance1/remove_syscall_indirection
Arm64: Removes a vtable indirection in syscalls
2024-01-04 18:19:40 -08:00
Ryan Houdek 4b3792196f Merge pull request #3303 from Sonicadvance1/initial_runtime_longmode_switch
OpcodeDispatcher: Initial support for runtime long-mode switch
2024-01-04 18:17:54 -08:00
Ryan Houdek c333aac4f9 Merge pull request #3354 from Sonicadvance1/uprev_kernel_2
Linux uprev to v6.6
2024-01-03 14:25:13 -08:00
Ryan Houdek db7d7a6bd7 Merge pull request #3349 from Sonicadvance1/revert_frontend_ownership
Revert "FEXLoader: Moves thread management to the frontend"
2024-01-03 14:25:04 -08:00
Alyssa Rosenzweig 04a88ed3ab Merge pull request #3353 from Sonicadvance1/public_interface_cleaning
FEXCore interface cleaning
2024-01-03 15:14:54 -04:00
Alyssa Rosenzweig 9da08b40bd Merge pull request #3344 from Sonicadvance1/xbyak_upstream
Externals: Update xbyak to v7.02 and switch away from fork
2024-01-03 15:13:58 -04:00
Alyssa Rosenzweig 5467c3e478 Merge pull request #3357 from Sonicadvance1/remove_non_sra
FEXCore: Removes SRA option, it's now permanently enabled
2024-01-03 15:10:04 -04:00
Ryan Houdek 9841983955 Merge pull request #3358 from wannacu/main
JIT: Fixes broken register in VTBX1
2023-12-30 00:44:09 -08:00
wannacu 4e7bab849c JIT: Fixes broken register in VTBX1
If the Dst register is allocated as VectorIndices or VectorTable,
using Dst as an operand to perform the tbx operation will result in an error.
For example:
%131(FPR0) i128 = LoadNamedVectorIndexedConstant u8:Tmp:RegisterSize, #0x6, #0xaa0
%132(FPR0) i128 = VTBX1 u8:Tmp:RegisterSize, %129(FPRFixed6) i32v4, %126(FPRFixed10) i16v8, %131(FPR0) i128
Since the tbx instruction's destination register is also the original operand,
this is consistent with the semantics of VTBX1. Therefore,
directly using VectorSrcDst as the destination operand for the tbx instruction is safe.
2023-12-29 16:18:40 +08:00
Ryan Houdek d098545c20 FEXCore: Removes SRA option, it's now permanently enabled 2023-12-28 18:28:02 -08:00
Ryan Houdek 5358af7794 Revert "FEXLoader: Moves thread management to the frontend"
This reverts commit 58f2693954.
2023-12-27 04:33:50 -08:00
Ryan Houdek eea2e7bb57 Merge pull request #3351 from Sonicadvance1/remove_context_lock
FEXCore: Removes context wide and map lookup
2023-12-27 00:59:22 -08:00
Ryan Houdek 25bcddf3a5 FEXCore: Removes context wide and map lookup
While locking a shared_lock and doing an empty table lookup is fairly
fast, just remove them from the hot path entirely if no custom IR
handlers are installed.

This is only used for our IRLoader, which is losing its importance
significantly and should probably be removed anyway.
2023-12-26 11:11:44 -08:00
Ryan Houdek f785b38e4d Merge pull request #3352 from Sonicadvance1/remove_irloader
Removes IRLoader, unittests, and public interface
2023-12-26 11:08:26 -08:00
Ryan Houdek 0071c1bda2 Merge pull request #3350 from Sonicadvance1/minor_opt
FEXCore: Optimize HostFeatures and CPUID feature calculation
2023-12-26 09:01:58 -08:00
Ryan Houdek 058e691ef1 Merge pull request #3348 from Sonicadvance1/disable_sha
Support disabling SHA in CPUID
2023-12-26 08:24:59 -08:00
Tony Wasserka d806db53ec Merge pull request #3318 from neobrain/feature_thunk_layout_wrappers
Library Forwarding: Emit layout helpers to allow repacking struct data
2023-12-26 16:25:53 +01:00
Tony Wasserka bcd0efa724 Thunks: Build fix for clang versions older than 15
Returning incomplete types from deleted functions is valid C++, but clang
did not support it until version 15:
https://github.com/llvm/llvm-project/issues/52802
2023-12-26 16:02:08 +01:00
Tony Wasserka 48c2e0689a Thunks/gen: Specialize layout wrappers for pointer types
Pointer types inherently cause data layout compatibility issues, so they're
worth special-casing here. The wrappers will type-pun pointers to 32-bit or
64-bit integers (matching the guest architecture) to avoid direct host-side
use of guest pointers without consideration.
2023-12-26 16:02:08 +01:00
Tony Wasserka 4b09a6bee1 Thunks/gen: Implement host_layout->guest_layout conversion
This enables use of guest_layout for return values.
2023-12-26 16:02:08 +01:00
Tony Wasserka af645cb750 Thunks/gen: Fill guest_layout stub and implement conversion to host_layout
The guest_layout wrapper provides an architecture-agnostic representation of
the guest data layout of each struct used in a thunked library. A constructor
is added to host_layout to allow conversion of the data to the host layout.

For types that are already fully compatible, both layout wrappers are simple
type aliases to minimize overhead.
2023-12-26 16:02:07 +01:00
Ryan Houdek b8b9b2f1a7 Linux: Update syscalls for v6.6 2023-12-25 08:40:46 -08:00
Ryan Houdek f02bd5c9a6 Linux/drm: Update to v6.6 2023-12-25 08:28:45 -08:00
Ryan Houdek 2b39813581 Linux: Implement v6.5 syscall that was added
Fairly easy since the arguments can just be passed through.

The rest of the v6.5 changes don't seem to be anything worth caring
about.
2023-12-25 08:17:43 -08:00
Ryan Houdek 923c53c4ea Linux: New socket options 2023-12-25 08:11:15 -08:00
Ryan Houdek 66658d82c6 Linux: Uprev kernel to 6.4
Added a new prctl that we need to emulate.
Some ioctls have changed that we don't really care about or will
naturally work.
2023-12-25 07:59:03 -08:00
Ryan Houdek b115c144fb FEXCore: Removes NetStream from public API
Only used by GDBServer.
NFC.
2023-12-25 07:07:17 -08:00
Ryan Houdek d8f20751fe FEXCore: Moves IREmitter from the public API to backend
No functional change
2023-12-25 07:00:29 -08:00
Ryan Houdek 1977747fc2 Removes IRLoader, unittests, and public interface
This unit test hasn't really served any purpose for a while now and
mostly just causes pain when reworking things in the IR.

Just remove the IRLoader, its unit tests, the github action steps and
the public FEXCore interface to it. Since it isn't used by anything
other than Thunks.

Also moves some IR definitions from the public API to the backend.
2023-12-25 07:00:29 -08:00
Ryan Houdek 257016bf12 FEXCore: Moves BucketList out of public API
NFC
2023-12-25 06:58:22 -08:00
Ryan Houdek 69d65fba4a FEXCore: Removes unused SyscallVisitor
This was expected to be part of the syscall optimizations we did but
ended up getting manifested in a different way. Remove it.
2023-12-25 06:42:11 -08:00
Ryan Houdek bce694ebb5 FEXCore: Moves BitUtils to FHU
No functional change
2023-12-25 06:38:51 -08:00
Ryan Houdek 5d37d5db1a FEXCore: Optimize HostFeatures and CPUID feature calculation
Need #3348 merged first.

As I was casually thinking, this code made me realize that it was quite
branch heavy and could likely be optimized to logic.

The previous code generated some fairly nasty branch heavy code. This
can be optimized to be branchless and take roughly five instructions
per flag. Using a bitfield for each feature would turn each calculation
in to 3-4 instructions but that seems overkill.

Very minor thing.
2023-12-25 04:58:15 -08:00
Ryan Houdek 4d109c9ce0 Config: Fixes parsing strenum inside of json files
This wasn't wired up before.
2023-12-23 22:32:59 -08:00
Ryan Houdek db9b326534 FEXCore: Support disabling CPUID features based on config
Need to be able to disable sha by config.
2023-12-23 22:32:29 -08:00
Ryan Houdek 21ae8336cd Update issue templates
Fex doesn't support x86 anymore
2023-12-22 18:03:11 -08:00
Tony Wasserka c6f8901c16 Thunks/gen: For repackable/compatible structs, use a richer host_layout/guest_layout interface 2023-12-22 20:55:15 +01:00
Tony Wasserka a797699d62 Thunks/gen: Sort types by dependency before emitting data
This allows future changes to emit interdependent helper structures in the
same order.

The sort algorithm was chosen for simplicity rather than performance. It's
fast enough in practice even for APIs as large as Vulkan.
2023-12-22 20:55:15 +01:00
Tony Wasserka 36c524a021 Thunks: Remove callback_guest and fex_guest_function_ptr
These are superseeded by ptr_passthrough and the guest_layout wrapper.
2023-12-22 20:55:15 +01:00
Tony Wasserka dbf33523bd unittests/thunks: Fix build on newer libclang versions 2023-12-22 20:55:15 +01:00
Ryan Houdek de2cd469a2 Merge pull request #3346 from Sonicadvance1/remove_32bit_allocator
FEX: Removes legacy kernel 32-bit allocator
2023-12-22 10:35:49 -08:00
Ryan Houdek 1c34b25538 FEX: Removes legacy kernel 32-bit allocator
We only used this so that our Xavier CI system which were running old
kernels could run unit tests. We have now removed the Xaviers from CI
and this is no longer necessary.

Stop pretending that we support kernels older than 5.0 and allowing this
fallback.

The 32-bit allocator is still used for the MAP_32BIT mmap flag, so the
load bearing code can't be fully removed. Just remove the config and the
frontend things using it.
2023-12-21 06:21:01 -08:00
Ryan Houdek 38ad3f0e05 FEXCore: Pass thread object to HandleUnalignedAccess
Currently no functional change but public API breaks should come early.

The thread state object will be used for looking up thread specific
codebuffers in the future when we support MDWE with code mirrors.
2023-12-21 01:55:25 -08:00
Ryan Houdek 266f7feecb Arm64: Removes a vtable indirection in syscalls
We can safely call virtual functions through the JIT with a little bit
of work.

FEX's JIT has quite a few steps before it gets to a syscall handler.

Before this commit:
JIT->static HandleSyscall->SyscallHandler::HandleSyscall->SyscallHandler

After this commit:
JIT->SyscallHandler::HandleSyscall->SyscallHandler

A bit hard to notice this when this interface can spin at 67-million
calls per second though.
2023-12-21 01:55:02 -08:00
Ryan Houdek f9902142f7 Utils: Add ability to get VTable entries to PMF helper
This will be useful to remove an indirection.
2023-12-21 01:55:02 -08:00
Ryan Houdek 175d407c9f InstCountCI: Update for segment changes 2023-12-21 01:54:19 -08:00
Ryan Houdek ae2f98e017 Mingw: Update for GDT 2023-12-21 01:54:19 -08:00
Ryan Houdek 9e5d7aa5fe OpcodeDispatcher: Initial support for runtime long-mode switch
This has the Frontend and OpcodeDispatcher select their operating mode
depending on the incoming code segment long-mode flag.

Adds some asserts since currently it is unexpected if the configuration
changes at runtime.

This is fairly straightforward for an initial setup but isn't fully
fleshed out.

Right now FEX's x86 tables aren't setup in a way to support choosing a
different instruction decoding depending on runtime operating mode
change, so that would break in interesting ways.

Primarily this just gets FEX setup to start piping the operating mode
through from the frontend to the backend. This is a long term task, so
it is going to take a long time to iron out all the issues.
2023-12-21 01:54:19 -08:00
Ryan Houdek 8648fb1485 FEXCore: Accurately store segment descriptors
Previously we were only storing the 32-bit base address which isn't
actually how segment descriptors work.

In reality segment descriptors are 64-bit descriptors that are laid out
in a particular layout depending on the 4-bit type value. In reality we
only care about code and data segment layouts since the rest are
bonkers.

Describe these descriptors correctly and setup a default code descriptor
for the operating mode that FEX is starting in.
2023-12-21 01:54:18 -08:00
Ryan Houdek 8b24f7fc26 Externals: Update xbyak to v7.02 and switch away from fork
The last few patches we need have been upstreamed so we shouldn't need
our downstream fork anymore.
2023-12-21 01:52:05 -08:00
Ryan Houdek b1c3737aef Merge pull request #3342 from Sonicadvance1/fix_proton_3
Fixes Proton again
2023-12-20 23:15:10 -08:00
Ryan Houdek 00669a1c89 Merge pull request #3336 from Sonicadvance1/warn_on_mdwe
FEXCore: Warn if MDWE is set
2023-12-20 13:27:43 -08:00
Ryan Houdek 1cedc3d85a FEXCore: Warn if MDWE is set
This will result in FEX not being able to allocate executable memory.
We can use shared memory in the future to work around this but for now
we don't support that as a fix.
2023-12-20 13:19:28 -08:00
Mai 5e26b77a7c Merge pull request #3326 from Sonicadvance1/thread_frontend_ownership_part3
FEXLoader: Moves thread management to the frontend
2023-12-19 21:05:18 -05:00
Ryan Houdek 58f2693954 FEXLoader: Moves thread management to the frontend
Lots going on here.

This moves OS thread object lifetime management and internal thread
state lifetime management to the frontend. This causes a bunch of thread
handling to move from the FEXCore Context to the frontend.

Looking at `FEXCore/include/FEXCore/Core/Context.h` really shows how
much of the API has moved to the frontend that FEXCore no longer needs
to manage. Primarily this makes FEXCore itself no longer need to care
about most of the management of the emulation state.

A large amount of the behaviour moved wholesale from Core.cpp to
LinuxEmulation's ThreadManager.cpp. Which this manages the lifetimes of
both the OS threads and the FEXCore thread state objects.

One feature lost was the instruction capability, but this was already
buggy and is going to be rewritten/fixed when gdbserver work continues.

Now that all of this management is moved to the frontend, the gdbserver
can start improving since it can start managing all thread state
directly.
2023-12-19 17:43:04 -08:00
Mai b4b8e81f24 Merge pull request #3321 from Sonicadvance1/thread_frontend_ownership_take2
FEXCore: Changes ParentThread ownership from the CTX to the frontend, take 2
2023-12-19 20:37:59 -05:00
Mai 82ce76b1d3 Merge pull request #3325 from Sonicadvance1/move_pmf
Dispatcher: Convert GetCompileBlockPtr to using PMF helper
2023-12-19 20:33:53 -05:00
Ryan Houdek a8f797d36b Dispatcher: Convert GetCompileBlockPtr to using PMF helper
This was older code that was written before the PMF helper was
available.
Switch it over.
2023-12-19 17:20:37 -08:00
Ryan Houdek 93ec676ce8 Merge pull request #3340 from Sonicadvance1/exitfunctionlink_data
FEXCore: Describe exit function linking object with a structure
2023-12-19 16:17:21 -08:00
Ryan Houdek 362cccc210 Syscalls: Fixes invalid register invalidation
FEX was incorrectly invalidating RBX and RBP when a new
thread/clone/fork was created. Only RAX should be set to zero to
signify that it is the child process.

I tried to find some reference as to why this was being done in FEX, and
I seem to recall some code somewhere that was setting up this state.

Sadly I can't seem to find any reference to this being required now by
the linux kernel when it creates new processes, so maybe I was just
gaslighting myself.

Fixes crash in bubblewrap where it was expecting a valid RBP to exist
after it does a clone syscall.

I have unit tests coming for this but it is uncovering some more bugs in
FEX's thread handling. Some of which are getting fixed with the current
PRs that are moving the thread handling to the frontend.

But since proton is broken now, this needs to get in.
2023-12-19 16:01:21 -08:00
Ryan Houdek c45dc6dd48 Syscalls: Fixes StackSize argument usage
This was left over and was using the wrong argument.
This would cause the stack for the child thread/process to be at the
start instead of the end.

Fixes a crash in bubblewrap.
2023-12-19 15:56:43 -08:00
Mai 3d2cbc5d08 Merge pull request #3317 from Sonicadvance1/fix_imul_flags2
OpcodeDispatcher: Fixes flags generation in imul
2023-12-19 11:43:21 -05:00
Mai 81c85d73b2 Merge pull request #3330 from Sonicadvance1/optimize_sib_addr_calc
OpcodeDispatcher: Optimize SIB addr calculation
2023-12-19 11:41:42 -05:00
Mai 5b4e9c6907 Merge pull request #3323 from Sonicadvance1/remove_unused_check
Dispatcher: Removes unused asserting CompileBlock function
2023-12-19 11:38:57 -05:00
Ryan Houdek aa2e8704bc FEXCore: Changes ParentThread ownership from the CTX to the frontend, take 2
Similar to #3284 but works around some of the bugs that one introduced.

This is the minimal amount of changes to move the ownership from FEXCore
to the frontend. Since the frontends don't yet have a full thread state
tracking, there is an opaque pointer that needs to be managed.

In the followup commits this will be changed to have the syscall handler
to be the thread object manager.
2023-12-18 14:54:07 -08:00
Ryan Houdek cf86ae6b65 FEXCore: Describe exit function linking object with a structure
Instead of just poking raw uint64_t data values, describe it with a
struct.

This will be a read-only in the future.
2023-12-18 13:31:53 -08:00
Ryan Houdek 68d6cf5f14 Merge pull request #3332 from Sonicadvance1/move_testharness_runner
TestHarnessRunner: Move to its own tool folder
2023-12-18 12:30:31 -08:00
Ryan Houdek c7bc15785b FEXLoader: Moves mingw check up one level
Now that only FEXLoader is under the FEXLoader folder, the mingw build
check can be moved up one level.

No functional change, just moving the check and re-aligning. Might be
good to view without whitespace changes.
2023-12-18 12:16:11 -08:00
Ryan Houdek c3d2d01c1f TestHarnessRunner: Move to its own tool folder
Needs #3331 merged first

No functional change, just code movement.
2023-12-18 12:16:11 -08:00
Ryan Houdek 9dda960529 Merge pull request #3331 from Sonicadvance1/move_irloader
Tools: Moves IRLoader to independent folder
2023-12-18 12:15:15 -08:00
Ryan Houdek f2bcc14cda Tools: Moves IRLoader to independent folder
Which requires moving LinuxEmulation to its own independent folder as
well. Since both IRLoader and FEXLoader rely on it.

No functional change, just moves the the code around.
2023-12-18 12:04:28 -08:00
Ryan Houdek 86654907bf Merge pull request #3334 from Sonicadvance1/remove_old_x86jit_references
FEXCore: Removes stale references to x86 JIT
2023-12-18 04:15:29 -08:00
Ryan Houdek 4333261639 Merge pull request #3329 from Sonicadvance1/vdso_fix_symbols
VDSO: Fixes symbol fetching for a few symbols
2023-12-18 04:15:10 -08:00
Ryan Houdek 12b72f908b Merge pull request #3335 from Sonicadvance1/remove_internalthreadstate_header
FEXCore: Removes old InternalThreadState header
2023-12-18 04:14:57 -08:00
Ryan Houdek f5997a084e Merge pull request #3333 from Sonicadvance1/remove_cpuid_init
CPUID: Removes Init and just uses constructor
2023-12-18 04:14:36 -08:00
Ryan Houdek 6c8a54ff84 FEXCore: Removes old InternalThreadState header
This was a temporary header to help with when this header was migrated
to our public API headers.

It's temporary nature is no longer necessary, just get rid of it.
2023-12-15 18:51:25 -08:00
Ryan Houdek bcc2901d7f FEXCore: Removes stale references to x86 JIT
It doesn't exist anymore.
2023-12-15 18:46:57 -08:00
Ryan Houdek 358bbb51ff CPUID: Removes Init and just uses constructor
No need to wait for initialization on for this anymore.
Ever since Init was refactored to do basically no work, this hasn't been
necessary.

CPUID does need to still be initialized after HostFeatures though, so
need to ensure correct member ordering there.
2023-12-15 18:43:23 -08:00
Ryan Houdek f2da70c7e7 InstCountCI: Update for SIB optimization
Only LEA uses SIB scale*index+base.
2023-12-15 13:11:42 -08:00
Ryan Houdek 1a2f41922c OpcodeDispatcher: Optimize SIB addr calculation
When the address calculation for SIB has both index and base then we can
optimize this to an add with a shifted register. This will convert a
three instruction sequence in to one instruction in most cases.
2023-12-15 13:08:46 -08:00
Ryan Houdek 0ede707e0b IR: Adds support for AddShift IR op
This matches x86 SIB's operation of `scale * index + base`
2023-12-15 13:08:06 -08:00
Ryan Houdek c4c31d4e80 VDSO: Fixes symbol fetching for a few symbols
When originally written I was testing symbol loading on an x86 host
assuming that the vdso symbol names were the same between x86 and
aarch64.

Turns out this is not true and aarch64 uses `__kernel_` prefix instead
of x86's `__vdso_`.

AArch64 VDSO only provides us with four symbols right now
- __kernel_rt_sigreturn
- __kernel_gettimeofday
- __kernel_clock_gettime
- __kernel_clock_getres

This means that currently AArch64 is missing getcpu and time.
time is unlikely to ever be implemented so the vdso scanning for that is
unlikely to ever get a pointer and will need to go down the glibc path.

getcpu is likely to get an implementation at some point. There's even an
old patch series that implemented it[1]. Currently the patch has been
dropped on the floor for whatever reason but hopefully they will come
back to it.

Benchmarks on Cortex-X1C:
- Native AArch64 clock_gettime bench
   - VDSO: 33.3ns per call
   - glibc (which uses vdso): 33.3ns per call
   - Roughly equivalent
- Emulated x86_64 clock_gettime bench
   - Prior to fix:
      - 49ns per call for both VDSO and glibc
   - After fix:
      - VDSO: ~46ns per call
      - glibc: ~49ns per call
- Emulated i386 clock_gettime bench
   - Only tested after fix
   - VDSO: 50ns per call
   - glibc: 65ns per call

The only way to improve this further would be to optimize thunks to push
certain function signature arguments in registers instead of stack.
Which is a future optimization as time goes on.

[1] https://patchwork.kernel.org/project/linux-kselftest/list/?series=335203
2023-12-15 10:56:58 -08:00
Ryan Houdek 12923ba1b7 Merge pull request #3322 from Sonicadvance1/remove_unused_exithandler
PassManager: Removes unused exit handler
2023-12-14 01:55:06 -08:00
Ryan Houdek bd13052708 Merge pull request #3315 from Sonicadvance1/clone_fix_allocation
Linux: Adjust when clone allocates stack memory
2023-12-12 07:31:04 -08:00
Ryan Houdek ec89a00a92 Merge pull request #3320 from Sonicadvance1/consteval_x86_tables
X86Tables: Converts tables to be mostly consteval
2023-12-11 23:03:50 -08:00
Ryan Houdek e657a27607 Dispatcher: Removes unused asserting CompileBlock function
While we were calling this function, its asserting nature hasn't been
used for a long time.

This used to trigger more frequently when CompileBlock would fail to
compile code, either due to not being able to decode an instruction or
hitting an instruction that FEX doesn't understand.

When these cases are hit today we still generate code blocks which
generate SIGILL. This means that this code was actually never hit.

Completely remove this function and have the JIT's dispatcher call the
CompileBlock function directly. Signature is slightly different since we
need to set x3 to be 0.
2023-12-11 17:20:51 -08:00
Ryan Houdek 8bb5462554 PassManager: Removes unused exit handler
git blame shows that 718b3e6b4c added this
handler.

It doesn't explain why this was desired but it was never wired up to
anything. Just remove it.
2023-12-11 17:14:41 -08:00
Ryan Houdek 98f21a2a28 X86Tables: Converts tables to be mostly consteval
Reduces the ELF's VM size from 9.8MB down to 9.37MB and should reduce
initialization time a smidge.

Slammed this out while waiting for other PRs to get reviewed.
2023-12-11 10:03:52 -08:00
Ryan Houdek 0a4e064da4 Merge pull request #3296 from Sonicadvance1/mov_thread_creation
FEXCore: Moves OS thread creation to the frontend
2023-12-11 06:35:26 -08:00
Ryan Houdek 5660065eea FEXCore: Moves OS thread creation to the frontend
Fairly lightweight since it is almost 1:1 transplanting the code from
FEXCore in to the SyscallHandler's thread creation code.

Minor changes:
- ExecutionThreadHandler gets freed before executing the thread
   - Saves 16-bytes of memory per thread
- Start all threads paused by default
   - Since I moved the code to the frontend, I noticed we needed to do
     some post thread-creation setup.
   - Without the pause we were racing code execution with TLS setup and
     a few other things.
2023-12-11 06:22:50 -08:00
Ryan Houdek 131bf48f4b Linux: Adjust when clone allocates stack memory
clone needs to allocate memory before the fork locks are held, otherwise
they will hang forever waiting on a locked mutex.

This wasn't previously seen since nothing was using fork with clone3.
2023-12-11 06:21:14 -08:00
Ryan Houdek a1cf14f2d9 Merge pull request #3319 from Sonicadvance1/fix_gdbserver_crash
GdbServer: Fixes crash on gdb detach
2023-12-11 06:19:40 -08:00
Ryan Houdek 499a40ecb5 GdbServer: Fixes crash on gdb detach
GdbServer object was getting deleted before the socket thread was
shutdown which was causing a crash on detach.

Now on destructor it will wait for the thread to thread to exit.
2023-12-08 06:03:58 -08:00
Ryan Houdek 7524029a06 Merge pull request #3294 from Sonicadvance1/mov_xid_check
FEXCore: Moves XID check to the frontend
2023-12-07 01:17:45 -08:00
Ryan Houdek fe3df001e2 InstCountCI: Update for instruction changes. 2023-12-07 01:10:09 -08:00
Ryan Houdek 2053c712a5 unittests/ASM: Adds test to test imul flag calculation
This was incorrect before.
2023-12-07 01:09:39 -08:00
Ryan Houdek 5c6f229e76 OpcodeDispatcher: Fixes flags generation imul
On overflow with 32-bit we weren't setting the flags correctly.
2023-12-07 01:08:02 -08:00
Ryan Houdek acdb4c7061 IR: Adds support for {S,U}Mull
Lets us do a 32-bit multiply returning a 64-bit result, signed and
unsigned.
2023-12-07 01:06:58 -08:00
Ryan Houdek b613576ac4 Merge pull request #3314 from Sonicadvance1/more_installscript
Scripts: More changes to InstallFEX script
2023-12-06 23:57:19 -08:00
Ryan Houdek 8f52d8e719 Scripts: More changes to InstallFEX script
A dictionary lookup was only ever returning the .ero result since we
were adding multiple values to the same key. Adds them as a list now and
walks them.

Additionally if we forget to add a new version again, have a fallback to
try checking using our file format layout.

Fixes an issue where a user running 23.10 was faulting the python
script.
2023-12-06 23:50:43 -08:00
Alyssa Rosenzweig 0fe5e3d1e7 Merge pull request #3313 from alyssarosenzweig/opt/andn-mask
rm andn masking
2023-12-06 12:21:24 -04:00
Ryan Houdek 26c9d5d427 Merge pull request #3310 from Sonicadvance1/disable_clear_sighand
FEXLoader: Temporarily disable CLONE_CLEAR_SIGHAND
2023-12-06 00:04:16 -08:00
Ryan Houdek 60b0852cde Merge pull request #3309 from neobrain/fix_thunks_32bit_addresses
Thunks: Add workarounds for pointers not readable by 32-bit guests
2023-12-05 23:52:59 -08:00
Ryan Houdek 3a5ac39bb1 Merge pull request #3312 from neobrain/fix_format_strings
FEXLoader: Fix incorrect format strings
2023-12-05 23:51:00 -08:00
Ryan Houdek 301310329b FEXLoader: Temporarily disable CLONE_CLEAR_SIGHAND
This flag breaks FEX heavily for now.
glibc 2.38 started using this flag as an optimization for posix_spawn.
It will fall back to a "non-optimized" implementation if the clone
syscall returns EINVAL. For now do this while we investigate a more
proper implementation.

Should be backported to 2312.1.
2023-12-05 23:46:48 -08:00
Ryan Houdek 86c6ca3049 Merge pull request #3307 from neobrain/fix_vulkan_query_through_instance
Thunks: Allow querying customized functions through vkGetInstanceProcAddr
2023-12-05 23:40:03 -08:00
Ryan Houdek eb5cf1a667 Merge pull request #3311 from Sonicadvance1/update_install_script
Scripts: Updates InstallFEX with supported Ubuntu versions
2023-12-05 19:35:06 -08:00
Alyssa Rosenzweig f515b1eeb1 Merge pull request #3308 from neobrain/fix_armops_warning
Arm64/BranchOps: Fix unused-variable warning
2023-12-05 09:07:07 -04:00
Alyssa Rosenzweig 924cfb5429 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-12-05 09:05:25 -04:00
Alyssa Rosenzweig 2a2c389be6 OpcodeDispatcher: rm masking in 32-bit andn
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-12-05 09:04:32 -04:00
Tony Wasserka aaef83b777 FEXLoader: Fix incorrect format strings 2023-12-05 12:04:16 +01:00
Ryan Houdek 29d82cda48 Scripts: Updates InstallFEX with supported Ubuntu versions
Forgot to update this as our supported Ubuntu distros changed.
2023-12-05 02:44:17 -08:00
Tony Wasserka 9f721e96db Thunks: Work around argument data on the host stack not being accessible by 32-bit guests 2023-12-05 11:32:09 +01:00
Tony Wasserka 281be3aeeb FEXLoader: Exhaust internal jemalloc allocations after 32-bit address space blocking 2023-12-05 11:27:39 +01:00
Tony Wasserka 5b87307452 FEXLoader: Cleanup 32-bit allocator setup 2023-12-05 11:24:51 +01:00
Tony Wasserka aebabf6ef2 Revert "FEXLinuxTests: Temporarily limit thunk test execution to 64-bit guests"
This reverts commit 21b6cccb4e.
2023-12-04 14:57:43 +01:00
Tony Wasserka 0d5fd5dbaf FEXLinuxTests/thunks: Disable tests involving pointers on 32-bit
Passing pointers across architecture boundaries is not yet supported.
Skipping these tests allows for the rest to run on 32-bit.
2023-12-04 14:57:43 +01:00
Tony Wasserka 3729b02255 Arm64/BranchOps: Fix unused-variable warning 2023-12-04 11:32:09 +01:00
Tony Wasserka d4e9ed8e61 Thunks/vulkan: Use string_view literals to express string comparisons
This reduces syntax noise as more functions will be added to this list.
2023-12-04 11:09:25 +01:00
Tony Wasserka c10402f4f9 Thunks/vulkan: Move custom impl matching to common function 2023-12-04 11:09:25 +01:00
Ryan Houdek 64276dbd0c Docs: Update for release FEX-2312 2023-12-03 17:59:59 -08:00
Alyssa Rosenzweig b619f381b3 Merge pull request #3306 from Sonicadvance1/move_remaining_nzcv
IR: Moves remaining NZCV operations to use DestSize
2023-12-03 21:06:22 -04:00
Ryan Houdek 3b0aff5fb9 IR: Moves remaining NZCV operations to use DestSize
Just like in #3305 but wasn't causing any known issues
2023-12-02 15:40:51 -08:00
Alyssa Rosenzweig a8ab8bbe8e Merge pull request #3305 from Sonicadvance1/fix_crash2
JIT: Fixes crash in TestNZ
2023-12-02 19:35:10 -04:00
Ryan Houdek 3e2ba6d835 unittests/ASM: Adds test for TestNZ bug
Would trigger the bug before it was fixed.
2023-12-02 14:57:57 -08:00
Ryan Houdek c6497fe32b JIT: Fixes crash in TestNZ
In some situations TestNZ is generated with a constant that is using a
constant that can't fit inside of the tst instruction.

This was found in libGLX with virgl, crashing invalid instruction
generation and crashing steamwebhelper
2023-12-02 14:57:57 -08:00
Ryan Houdek c8ef77c15f Merge pull request #3297 from Sonicadvance1/remove_getprogramstatus
FEXCore: Removes GetProgramStatus
2023-11-30 16:29:37 -08:00
Ryan Houdek b35fadf7e3 Merge pull request #3298 from Sonicadvance1/remove_initializecontext
FEXCore: Removes InitializeContext API
2023-11-30 16:29:30 -08:00
Ryan Houdek 250ffb6d23 Merge pull request #3299 from alyssarosenzweig/opt/bt
Optimize BT/BTC/BTS/BTR
2023-11-30 10:50:29 -08:00
Alyssa Rosenzweig f6b1434d63 Merge pull request #3304 from Sonicadvance1/fix_corruption
Dispatcher: Fixes corruption when spilling SRA registers
2023-11-30 08:51:56 -04:00
Ryan Houdek 6bbae69c75 Dispatcher: Fixes corruption when spilling SRA registers
These functions only want the GPRs returned for SRA. This is because the
signal handler needs this map to relation between x86 GPRs and AArch64
GPRs.

When we added AF and PF to the SRA array we accidentally started
returning two more GPRs to the frontend. This caused the signal
delegator to start corrupting the members after GPRs in FEX's CoreState.

Corrupting 16-bytes after the gregs[] array.
This included corrupting:
   - es_idx, cs_idx, ss_idx, ds_idx, gs_idx, fs_idx, _pad[]
2023-11-29 16:17:17 -08:00
Alyssa Rosenzweig d0f54bcb23 Merge pull request #3278 from Sonicadvance1/fix_oversights
FEXCore: Fixes passing arguments to ABI helpers
2023-11-29 17:17:39 -04:00
Alyssa Rosenzweig 470615b896 Merge pull request #3301 from Sonicadvance1/fix_sonicmania_32bit
InstCountCI: Moves Sonic Mania code to 32-bit file
2023-11-29 15:36:45 -04:00
Ryan Houdek b02ab8ee19 InstCountCI: Moves Sonic Mania code to 32-bit file
Sonic Mania is a 32-bit game, it shouldn't be running its assembly as
64-bit.
2023-11-29 11:25:32 -08:00
Alyssa Rosenzweig 5f6046be4c InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 14:35:32 -04:00
Alyssa Rosenzweig e923e83efb OpcodeDispatcher: fix nzcvdirty
lets us use flagm in cmpxchg.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 14:35:25 -04:00
Alyssa Rosenzweig e836e4212d InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 14:23:41 -04:00
Alyssa Rosenzweig 9417c93110 OpcodeDispatcher: remove outdated comment
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 14:23:41 -04:00
Alyssa Rosenzweig 068599b1ec OpcodeDispatcher: use size-appropriate alu in bt*
saves zero-extending move for 32-bit ops.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 14:22:21 -04:00
Alyssa Rosenzweig 7216415bfc OpcodeDispatcher: reorder flag calcs in bt*
saves big moves.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 14:22:21 -04:00
Alyssa Rosenzweig 3020626506 OpcodeDispatcher: unify bt/btc/bts/btr impls
they're all copypastes of each other, unify into one general "bit test & perform
action" template. this means most of the wins from the previous commits now
apply for bt* without more copypaste.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 14:22:21 -04:00
Alyssa Rosenzweig 0a79fa8d5d OpcodeDispatcher: remove masking for 32/64-bit bt
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 13:40:18 -04:00
Alyssa Rosenzweig f8380b9adb OpcodeDispatcher: use smaller shifts for BT
if the shift is < N, and we grab bit 0 after, we only need to consider <=N
bits of the source. this lets us use 32-bit lsr for 32-bit bt, which will
reduce masking in the next commit.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 13:40:18 -04:00
Alyssa Rosenzweig d898028bc3 OpcodeDispatcher: optimize BT with constant
use the rmif properly.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 13:35:23 -04:00
Ryan Houdek f090700184 FEXCore: Removes InitializeContext API
This isn't necessary anymore, just initialize everything on context
creation immediately. All use cases just called this immediately
afterwards.
2023-11-29 09:33:32 -08:00
Ryan Houdek 01d29dffb9 FEXCore: Removes GetProgramStatus
Split from #3284 without changing ownership semantics while I reduce the
debugging surface here.

Removes one usage of ParentThread from FEXCore. Which can be done since
it is no longer an opaque structure, we can read the StatusCode
directly.

No functional change.
2023-11-29 09:23:47 -08:00
Alyssa Rosenzweig 14ba64a22d OpcodeDispatcher: use rmif masking for bt
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 13:20:22 -04:00
Alyssa Rosenzweig 6716077cb6 OpcodeDispatcher: don't mask bt source
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-29 13:16:55 -04:00
Alyssa Rosenzweig 8892580c41 Merge pull request #3295 from Sonicadvance1/fix_missing_cond_arg
IRDumper: Fixes missing conditional name
2023-11-29 12:57:49 -04:00
Ryan Houdek 1b41304fc1 IRDumper: Fixes missing conditional name
When COND_AL was added it wasn't added to this helper. Since there is a
gap between the last condition, just early check the value.

Fixes reading beyond the end of the array
2023-11-29 08:39:35 -08:00
Ryan Houdek afebf73e8f FEXCore: Moves XID check to the frontend
The XID signal handler is OS specific and needs to be delegated to being
handled in the frontend. This only happens when we create a thread with
pthread rather than clone/fork/vfork.

The first created pthread will reset the XID signal handler in glibc,
which we need to check the first time this occurs. A little bit disjoint
while the actual thread creation still happens in the backend but will
come together once the thread creation gets moved to the frontend.

The only game I am aware of that requires this to work is SOMA since its
one of its middleware libraries tries asking for root privileges.

Retested the game to make sure it still works as expected.
2023-11-28 19:19:06 -08:00
Ryan Houdek 7de66ac3a4 Merge pull request #3286 from Sonicadvance1/remove_getcpustate
FEXCore: Removes Get/SetCPUState
2023-11-28 18:54:29 -08:00
Alyssa Rosenzweig 85a1c1ff25 Merge pull request #3281 from Sonicadvance1/optimize_constantdirection_memsetcpy
FEXCore: Optimize memcpy and memset when direction is compile time constant
2023-11-28 19:11:27 -04:00
Alyssa Rosenzweig 8e892ece59 Merge pull request #3277 from Sonicadvance1/remove_warnings_17
FEXCore: Removes FEX_PACKED from CPUState
2023-11-28 19:11:18 -04:00
Ryan Houdek aa1344aadd Merge pull request #3293 from Sonicadvance1/erofs_extract
FEXRootFSFetcher: Supports extracting erofs images
2023-11-28 11:04:49 -08:00
Ryan Houdek 3f02d7c665 Merge pull request #3289 from Sonicadvance1/move_debug_names
FEXCore: Moves debug strings to gdbserver
2023-11-28 11:04:15 -08:00
Ryan Houdek f328fca880 Merge pull request #3283 from Sonicadvance1/thread_frontend_pt1
FEXCore: Start changing how thread creation works
2023-11-28 11:03:52 -08:00
Ryan Houdek 47d79978ef FEXRootFSFetcher: Fixes curl downloading with spaces in the path
Since we're invoking curl directly, we don't need to wrap it in `sh -c`
with this function.

Fixes an issue where curl downloads to a non-escaped path weren't
working. Now they do.
2023-11-28 01:19:46 -08:00
Ryan Houdek 2e24f34a3f Merge pull request #3291 from Sonicadvance1/remove_tmpfile
EmulatedFiles: Stop relying on O_TMPFILE
2023-11-27 13:48:31 -08:00
Ryan Houdek 6e8af295c5 Merge pull request #3290 from Sonicadvance1/move_signaldelegator
FEXCore: Moves more SignalDelegator functions to the frontend
2023-11-27 13:46:11 -08:00
Ryan Houdek bba156a3c1 FEXRootFSFetcher: Supports extracting erofs images
As I was poking around erofs-utils documentation, I found out that
fsck.erofs actually provides an option for extracting erofs images
without using fuse.

This finally puts the erofs handling on feature parity with squashfs.
2023-11-27 13:26:56 -08:00
Ryan Houdek 8015ce2099 Merge pull request #3292 from Sonicadvance1/remove_debug_x86_table
FEXCore: Removes x86 DebugInfo table
2023-11-27 13:13:18 -08:00
Ryan Houdek 1153c1a538 EmulatedFiles: Stop relying on O_TMPFILE
O_TMPFILE has a few minor problems that I have been thinking about for a
while. I just recently got reminded about this and remembered that most
problems get resolved by using memfd_create.

- O_TMPFILE is only supported on some filesystems.
   - Supported filesystem must be the one mounted to the pathname being
     opened.
   - Only a minor inconvenience as tmpfs and all related filesystems
     support this.
   - An inode is actually created on whatever filesystem is backing the
     folder.
- `/tmp/` must exist as a directory
   - If this folder happened to not be mounted then these temporary
     files wouldn't have been created.
   - memfd_create doesn't have a folder that needs to exist.
- We were leaving the files open as read/write
   - While we were rewinding the file offset, an misbehaving application
     could have wrote garbage to the temp file.
   - memfd sealing allows us to open the FD as RW and then seal its
     capabilities, making it a read-only FD.
- We were leaking FDs opened with O_CLOEXEC
   - We could have just opened the O_TMPFILE with O_CLOEXEC
   - memfd also just supports this flag, so use it.
   - No real issues, just nice to be sanitary here.

Overall this doesn't really change any behaviour, but it is nice to
cleanup some of the edges there.
2023-11-27 13:10:48 -08:00
Ryan Houdek a47b3cccb8 FEXCore: Moves debug strings to gdbserver
These are only used by gdbserver for filling out its XML data structures
so just remove them from FEXCore.

Also fixes the ordering on RegNames to match the definition of the enum
class definition in CoreState. This has been out of correct order since
we reordered registers months ago.
2023-11-27 13:08:37 -08:00
Ryan Houdek 2070056d16 FEXCore: Moves more SignalDelegator functions to the frontend
As we are moving more and more OS specific code to the frontend, this is
another set of functions that can be moved to FEXLoader from FEXCore.

No functional change here, only code moved from protected to private and
to FEXLoader's SignalDelegator.

Once more thread handling is moved to the frontend we can move even more
out of FEXCore. As follows:
- CheckXIDHandler can get moved.
  - First pthread FEX makes would just call this.
- Register/UnregisterTLSState
  - This can happen in the clone/thread handler once the frontend
    handles it.

This leaves very little in the backend and is mostly an interface for
passing signal data to the frontend that it needs once a signal has
occured.
It additionally also is used for `SignalThread`.
2023-11-27 12:59:46 -08:00
Ryan Houdek e227f1343f FEXCore: Start changing how thread creation works
The frontend needs to be in control of how threads are created. This is
inherent to the fact that OS threads are OS specific. We currently have
this weird split that when initializing the FEXCore context, we create a
parent thread at all times.

This does some initial cleanup that gets the core initialization nearly
decoupled.
2023-11-27 12:49:41 -08:00
Ryan Houdek 3c7335713d Merge pull request #3287 from Sonicadvance1/remove_config_threads
Config: Removes Threads option
2023-11-27 12:46:14 -08:00
Ryan Houdek bdf4089264 Merge pull request #3285 from Sonicadvance1/remove_getexitreason
FEXCore: Removes GetExitReason
2023-11-27 12:45:55 -08:00
Ryan Houdek b027113998 Merge pull request #3280 from Sonicadvance1/rpres_afp_requirement
FEXCore: Disables RPRES until AFP is audited and enabled
2023-11-27 12:45:39 -08:00
Ryan Houdek 389c6b11dd Merge pull request #3288 from Sonicadvance1/fix_dungeon_defenders
FEXCore: Fixes imul returning garbage data
2023-11-26 08:08:28 -08:00
Ryan Houdek fa5d9dc3b7 FEXCore/X86Tables: Shifts register down to offset zero
Flag position zero wasn't being used. Fill the hole.
Keeps the two overlay flags next to each other, moves the pop flag back
towards the end.
2023-11-25 17:10:23 -08:00
Ryan Houdek cb56728e57 FEXCore/X86Tables: Removes unused supports REP flag
This flag was being set in the tables but was actually unused.
2023-11-25 16:54:51 -08:00
Ryan Houdek b89c3a4573 FEXCore: Removes x86 DebugInfo table
This has long since been unused. Originally implemented for some fuzzing
tests but has been abandoned and that should likely be implemented some
other way.
2023-11-25 16:50:24 -08:00
Ryan Houdek 0cc11108ba InstcountCI: Update for changed codegen. 2023-11-24 14:15:06 -08:00
Ryan Houdek a7caf83022 FEXCore: Fixes imul returning garbage data
When a 32-bit imul was being executed it had a chance of returning
garbage data in the upper 32-bits of the 64-bit result.
While this didn't typically cause problems, this gets exacerbated from
32-bit applications executing multiplies for address calculations.

A combination of commits 7146691360 and
d01b457727 exposed this problem where
previously there would be multiple moves between the calculation and
data use which would have zero'd the upper bits for us previously.

Now that we are no longer doing that, we need to make sure the opcode
dispatcher doesn't generate broken code instead.

Fixes Dungeon Defenders, which hasn't worked since FEX-2308.

Adds an ASM test that ensures we don't break it again.
2023-11-24 14:04:05 -08:00
Ryan Houdek 053452c40c Config: Removes Threads option
This is an option that has been long overdue for removal. It's original
intention was primarily to lie to the guest application about the number
of cores in the system. This allowed us to say that the system was only
single threaded which worked around some threading bugs that we had
early on.

This is no longer the case and now it is a confusing remnant of the past
that people think they need to set. Incorrectly assuming that "0" by
default means that FEX is doing some sort of disabling of threading and
forcing all emulation down one CPU core. This is not the case and has
never been the case, so removing the option makes that idea go away.
2023-11-24 07:03:00 -08:00
Ryan Houdek d4361c87ae CPUID: Query and use physical cores upfront
Instead of using the config option, just query and use the result.
2023-11-24 07:02:30 -08:00
Ryan Houdek 6469eb7a0e Linux/EmulatedFiles: Stop using Threads config option
Now no longer potentially lying to the application about the number of
cores.
2023-11-24 07:01:56 -08:00
Ryan Houdek 13fbd0e802 Linux: Removes usage of CPU core config option
Stop lying to the application about getcpu, sched_getaffinity, and
sched_setaffinity.

- getcpu would wrap the cpu result modulo the count of cores
- sched_setaffinity wouldn't work at all
- sched_getaffinity lied and always reported full affinity of config
  option
2023-11-24 07:01:33 -08:00
Ryan Houdek e555a8f817 FEXCore: Removes Get/SetCPUState
Split off from #3282 to reduce burden.
We can read the data member directly now since it isn't opaque. In fact
we already do in the signal handlers. Removes these redundant helpers.

Removes one usage of ParentThread in FEXCore.
2023-11-23 07:55:06 -08:00
Ryan Houdek f9fb61cf1a FEXCore: Removes GetExitReason
Split off from #3282 to reduce burden.
We read the member directly now. Removes one usage of ParentThread
in FEXCore.
2023-11-23 07:50:23 -08:00
Ryan Houdek 43cf2e4e2c Merge pull request #3276 from bylaws/f1
Improvements for WOW64 compat with newer wine
2023-11-22 05:53:27 -08:00
Ryan Houdek 98f9a65202 Merge pull request #3275 from bylaws/x87fix
FEXCore: Work around broken preserve_all support in Windows clang
2023-11-22 05:53:14 -08:00
Ryan Houdek 8726c8fb73 Merge pull request #2691 from neobrain/refactor_scoped_signal_mask
ScopedSignalMask: Clean up API and use std::unique_lock/shared_lock
2023-11-19 04:53:05 -08:00
Ryan Houdek 27f3cb336f InstCountCI: Adds compile time constant memcpy/memset
To showcase the code.
2023-11-19 04:21:50 -08:00
Ryan Houdek aa3bacd938 FEXCore: Optimize memcpy and memset when direction is compile time constant
This usually happens on backwards memcpy where we know the direction of
the copy because the code will typically do as follows:

```
std
rep movsb
cld
```

This is because the direction flag is part of the ABI and needs to be
set back to the forward direction if it was modified.
This typically doesn't get picked up on forward copies because we won't
have visibility of a cld instruction in the block.

This optimization allows us to only emit half of the code for the memcpy
if it is a compile time constant.

There's definitely some future task that could assume forward direction
if unknown and recompile the code if the assumption has failed, but not
doing that here.
2023-11-19 04:17:10 -08:00
Ryan Houdek c71492ef32 FEXCore: Disables RPRES util AFP is audited and enabled
While auditing our JIT to see if we have any AFP issues I noticed this.

RPRES has a hard dependency that AFP exists in order to be used, we were
hitting a case where RPRES would be enabled, but AFP is disabled by
default.

RPRES only changes behaviour when FPCR.AH is set which requires AFP.
While this doesn't affect any hardware today, it likely will in the
future.
2023-11-19 03:24:19 -08:00
Ryan Houdek 70191f2d28 FEXCore: Fixes passing arguments to ABI helpers
Forgot to pass a few arguments through. Doesn't change behaviour since
we were always passing in TMP1 and FPRs = true but this is correct.
2023-11-19 02:15:45 -08:00
Ryan Houdek 93db8b7ca7 FEXCore: Removes FEX_PACKED from CPUState
Removes new warning from clang:
```
FEXCore/include/FEXCore/Core/CoreState.h:119:35: warning: not packing field 'DeferredSignalRefCount' as it is non-POD for the purposes of layout [-Wpacked-non-pod]
  119 |     NonAtomicRefCounter<uint64_t> DeferredSignalRefCount;
      |                                   ^
```

We already ensure sane data layout that this is unnecessary anyway.
2023-11-19 01:25:33 -08:00
Tony Wasserka d33b0cb9e3 SignalScopeGuards: Improve code gen for GuardSignalDeferringSectionWithFallback 2023-11-18 12:10:02 +01:00
Billy Laws 0806d4ec25 WOW64: Handle target threads where the JIT is uninitialised
Suspend may be called on a thread before it has finished WOW64 initialisation,
keep track of all initialized threads and fallback to direct
NtSuspendThread when this is the case.
2023-11-18 00:54:07 +00:00
Ryan Houdek 9b646746b2 Merge pull request #3274 from alyssarosenzweig/opt/pf
Arm64Emitter: Dedicate registers for PF/AF
2023-11-17 16:45:38 -08:00
Ryan Houdek 11993daec4 FEXCore: Hides eflags reconstruction information in the core
The frontend shouldn't need to know any information about how to
reconstruct eflags. Just give us the information we need and it'll work
out.
There are still some inherit limitations of this and some edge cases
that might give invalid data, but it is roughly as close as it was
before.

Just provide if the PC was in the JIT, the host GPRs, and the PState object from the signal
information and FEXCore does the rest.

We don't need to change the signature for `SetFlagsFromCompactedEFLAGS`
because during reloading of register state automatically does this for
us.
2023-11-17 20:38:42 -04:00
Billy Laws a78ffeeaba WOW64: Call Wow64ProcessPendingCrossProcessItems on syscalls
This is needed to handle code invalidation requests sent by external
processes.
2023-11-18 00:16:44 +00:00
Billy Laws 05b78339f6 FEXCore: Work around broken preserve_all support in Windows clang
While clang side code to support preserve_all on Windows is in place
(and thus there are no errors for using it), there are still some parts
missing on the LLVM side. [1]

[1] https://github.com/llvm/llvm-project/blob/2402b14046b9628b584c07789830d7ed481f7d74/llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp#L88
2023-11-17 22:13:39 +00:00
Alyssa Rosenzweig 365c221029 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig f60608a9c0 OpcodeDispatcher: allow garbage for 32bit inc/dec
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 153d871be2 OpcodeDispatcher: allow garbage for 32-bit cmp
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig b69f2d7773 InstCountCI: Start collecting hot blocks
Goal is to improve FEX performance with real world games. Let's start with
Sonic Mania. Two hot blocks from #2681.

I'm only bothering to track flagm for this because I'm super biased.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 09ffe7ef6b InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 3dfb94b524 OpcodeDispatcher: use axflag for x87-f64 fcmp
only 1 instr saved but meh.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig d1e43d94e9 OpcodeDispatcher: use axflag for fcmp faster
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 1b490e0e53 CodeEmitter: add ax/xaflag
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 094146d630 IR: add axflag
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 23c2a53683 OpcodeDispatcher: move fcmp flag fixup to dispatcher
simpler *and* much faster

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 82b7689ca4 OpcodeDispatcher: remove fcmp deferral
no longer load bearing, delete the abstraction.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 149f3e6f6d OpcodeDispatcher: rm flagsOp unused since select rework
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig cea551c2ac InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Alyssa Rosenzweig 2dcae23776 Arm64Emitter: Dedicate registers for PF/AF
Many flag-generating instructions like cmp need to save calculations for
deferred PF and AF flag calculation. Currently, they require a store per flag,
which is prohibitively expensive for hot instructions like cmp. By instead
pinning PF/AF temporary results to registers (x26/x27 by convention here), we
eliminate many stores altogether and turn the rest into zero-cycle moves (on
64-bit at least, this isn't optimal for 32-bit emulation due to CTX->GetGPRSize
shenanigans, need to check if this requirement can be lifted..).

To implement, we model as SRA and then the existing SRA code is able to generate
good code with little manual tuning. (Future work will get us to excellent code
with more tuning ;) ).

The tradeoff is reducing the working dynamic GPR set by 2 registers, which might
increase spilling in some cases. I think it's worth it in practice, though.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-17 17:37:24 -04:00
Tony Wasserka 92e4e75217 Merge DeferredSignalMutex.h and ScopedSignalMask.h into a single file
Using a single file makes sense now that the individual files are much
shorter and share common utility classes.
2023-11-17 10:56:34 +01:00
Tony Wasserka 5ca35bf77c ForkableMutex: Simplify WIN32 implementation 2023-11-17 10:56:34 +01:00
Tony Wasserka c956b82d27 ScopedSignalMask/DeferredSignalMutex: Clean up API and use std::unique_lock/shared_lock 2023-11-17 10:56:34 +01:00
Ryan Houdek 1c115096c4 Merge pull request #3268 from Sonicadvance1/more_gdbserver
More GdbServer improvements
2023-11-15 21:15:14 -08:00
Ryan Houdek c1d5fae018 Merge pull request #3273 from alyssarosenzweig/opt/shifts
Optimize shifts/rotates
2023-11-14 14:13:56 -08:00
Ryan Houdek 17d49fc00f Review comments 2023-11-14 12:54:54 -08:00
Alyssa Rosenzweig 0e1e4c16b1 Merge pull request #3271 from Sonicadvance1/optimize_sha
OpcodeDispatcher: Optimize three sha instructions
2023-11-14 16:48:46 -04:00
Alyssa Rosenzweig bec8e27b4f InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-14 08:40:41 -04:00
Alyssa Rosenzweig 56841f0e50 OpcodeDispatcher: avoid moves with 64bit imul
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-14 08:40:41 -04:00
Alyssa Rosenzweig 723146050b OpcodeDispatcher: allow garbage with multiplies
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-14 08:40:41 -04:00
Alyssa Rosenzweig 85b1aa4c2d OpcodeDispatcher: optimize mul flags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-14 08:40:41 -04:00
Ryan Houdek 0506369519 InstCountCI: Update for sha changes
Moves crypto instructions to files where crypto is enabled.
2023-11-13 18:38:02 -08:00
Ryan Houdek ba1632974e InstCountCI: Actually allow disabling crypto
Forcing it was required to work around initial simulator quirks
2023-11-13 18:38:02 -08:00
Ryan Houdek c69082b1a4 OpcodeDispatcher: Optimize three sha instructions
- sha1nexte
   - Takes advantage of sha1h if supported
   - Does the operation in a vector otherwise
- sha1msg2
   - Instead of dumping everything to GPRs, we can do this with vectors
   - Mostly matches ARM's sha1su1 instruction, but it is /just/
     different enough to be annoying.
- sha256msg1
   - Directly matches sha256u0
   - Leaves the previous implementation alone
2023-11-13 18:38:02 -08:00
Ryan Houdek 74b2548982 IR: Implement support for VUSHRAI IR op
This matches Arm64 usra semantics. This instruction is useful for
implementing vector element rotate.
2023-11-13 18:22:45 -08:00
Ryan Houdek f31656ec65 IR: Support sha1h and sha256u0
These match our needs so wire them up
2023-11-13 18:22:45 -08:00
Ryan Houdek e91420c405 HostFeatures: fixup SHA checks
- Simulator doesn't support SHA
- Use DisableCrypto option to disable sha as well
- Only enable SHA if ARM cpu supports both SHA1 and SHA2
2023-11-13 18:22:45 -08:00
Ryan Houdek 25df59a65d ArmEmitter: Fixes sha256u1 emitter
Noticed this was actually emitting sha256h2
2023-11-13 18:22:45 -08:00
Alyssa Rosenzweig 83fdd5720f IR: Add ccmn
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 22:05:02 -04:00
Ryan Houdek db63241fd4 Merge pull request #3270 from Sonicadvance1/x11_thunk_lock
Thunks/libX11: Change lock functions to nullptr by default
2023-11-13 17:29:43 -08:00
Alyssa Rosenzweig 89b00c89aa OpcodeDispatcher: optimize rcl 1-bit
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig d38917b5f0 OpcodeDispatcher: rm pointless constant
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 910e0242c1 OpcodeDispatcher: optimize rcr 1-bit
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 651b7bb75d OpcodeDispatcher: optimize RCL
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig c9f13ae1dd OpcodeDispatcher: optimize RCR the usual ways
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 862e575100 OpcodeDispatcher: avoid some ubfx for rcr with flagm
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 4669c4541c OpcodeDispatcher: don't zero for flagm ror
missed earlier in the PR, would be annoying to rebase in.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 769a8c41c4 OpcodeDispatcher: Branch over shift=0 flags
Not supposed to touch flags at all, so don't! instead of making a terrible mess
of csels. a lot less instructions, and probably faster because the branch should
be predicted correctly in practice in hot loops.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig bec9dba2b1 OpcodeDispatcher: use shifted xor + rmif for rotates
eliminates lots of Bfe on flagm.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 205ba2ea13 IR: add shifted xor
for rotates.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 2073f6d287 OpcodeDispatcher: don't zero nzcv for flagm shifts
Faster for flagm. would be slower for !flagm because bfi slowness...

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 0f25a960ee OpcodeDispatcher: remove bfe for small shl imm
We allow the garbage in flags calculation, it's ignored.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 282ed3e309 OpcodeDispatcher: optimize bsf/bsr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig cd031a7d38 OpcodeDispatcher: avoid some ubfx for flagm
Do the masking as part of the rmif, for free.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig e1885ed0bd OpcodeDispatcher: Use 64-bit ubfx
for larger shifts.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:21:01 -04:00
Alyssa Rosenzweig 4a31b619fa Merge pull request #3272 from alyssarosenzweig/instcountci/del-optimal
InstructionCountCI: Remove Optimal flags
2023-11-13 21:20:07 -04:00
Alyssa Rosenzweig bd4464bd5e InstructionCountCI: Remove Optimal flags
Instruction count CI has transformed the way we work on FEX… I love the system
and want to make it better. there’s one part of instruction count CI that isn’t
so lovable: the problematic “optimal” flag on instructions.

There are several issues with this flag, both philosophical and practical.

– it is tedious to update the optimal flag when making an implementation
optimal. The effect of that is discouraging people from making instructions,
optimal, or encouraging people to fail to update the flag, and dilute the value
of it. Either way, since we care far more about optimal implementations, then we
do about updating the flag, clearly we should prioritize the implementation and
not the flag. This issue was not obvious at the outset, when instruction count,
CI was introduced, and still quite small. The problem magnified when we started
duplicating instructions in bulk for different combinations of CPU features
(flagm, AFP, etc.) that intern multiplies the manual work required to update the
flags by the corresponding constant factor. if it comes down to a choice between
removing this extra coverage and removing the flag, I think we all agree that
removing the flag is the lesser evil.

– The definition of “optimal” is fundamentally problematic. I have often
improved the instruction count of an instruction that was already “optimal”.
This is all kinds of silly, and calls into question whether there’s any value
whatsoever in the existing classifications of the flag. Furthermore, it is often
unknowable, whether an implementation really is optimal. Is it possible to
implement BZHI (with flag calculations) in fewer than eight instructions? We
don’t know, and it’s silly to pretend that we do.

– as a consequence of the problematic definitions , there are so many errors in
both directions that I don’t think there’s much value in preserving the existing
classification at the expense of +progress. Being able to say “32% of
instructions are translated optimally” is neat, but it really doesn’t tell us
anything whatsoever when you dig a little deeper.

So, as the flag is misleading at best and perhaps harmful at worst, let’s remove
it and make the instruction count CI, more useful overall. let’s let the
expected count and the assembly speak for themselves, and cut away the chaff. if
we want a meaningless number to report to management, we can instead calculate
the average blowup factor ;-)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-13 21:14:05 -04:00
Ryan Houdek d907a7dc9f gdbServer: Adds some lldb specific query commands
Fairly small.
2023-11-13 15:18:38 -08:00
Ryan Houdek 732070f750 GdbServer: Try binding twice
If a bind fails then this is usually because PID reuse happened and the
unix domain socket path still existed on the filesystem. Alternatively
the process did an execve which left a dangling unix domain file.

Unlink the file and try again in these cases. It will almost certainly
work the second time.
2023-11-13 15:11:47 -08:00
Ryan Houdek 48442b6b03 GdbServer: Poll listen socket for connection
Instead of relying on sleeping in accept, use poll. This allows us to
early shutdown instead of leaving a thread hanging stuck in accept.

Doesn't change behaviour with FEX gdbserver waiting for attach, but in
the future when we all full process trees to have gdbserver running this
will be more important.
2023-11-13 15:11:47 -08:00
Ryan Houdek 7fdbe547a3 GDBServer: Set thread name
Makes it easier to pick it out of all the threads.
2023-11-13 15:11:47 -08:00
Ryan Houdek 59565b828d FEXServerClient: Expose the temp folder directly
Using the server mount folder works most of the time, but when running
under pressure-vessel this stacks directories in a weird way because the
mount folder has some tricks applied to it.

Expose the temp folder being used directly instead.
2023-11-13 15:11:47 -08:00
Alyssa Rosenzweig 1e2d059890 Merge pull request #3266 from alyssarosenzweig/opt/selects-and-more
Optimize selects & branches & more nzcv goodies
2023-11-13 14:20:42 -04:00
Ryan Houdek 0aa41908a2 Thunks/libX11: Change lock functions to nullptr by default
These are setup to be nullptr by default. Instead of providing no-op
lock instructions just have them be nullptr.

It's already part of the API that they need to be nullptr checked before
calling and this matches behaviour of the real libX11 library. Removes
some spam that is unnecessary.
2023-11-12 16:03:43 -08:00
Alyssa Rosenzweig b27ce3f79c InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:37:09 -04:00
Alyssa Rosenzweig 238e52f74a OpcodeDispatcher: Don't mask 32-bit bzhi either
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:36:46 -04:00
Alyssa Rosenzweig 9398b931fb Arm64Emitter: Handle 32-bit negatives
Noticed in the area.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:36:42 -04:00
Alyssa Rosenzweig b2a9785959 OpcodeDispatcher: optimize bzhi
Trickery to save an instruction :')

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 224a1f19a3 OpcodeDispatcher: improve bzhi flag gen
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 109c53f22b unittests: Add BZHI sign test
fails on main.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig e25849b2cb OpcodeDispatcher: fix BZHI flag calculation
needs SF.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 57978accc1 OpcodeDispatcher: Add heuristic to prefer rmif for InsertNZCV
Decent instcountci win.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig ff37177f4d IR: Remove AND(N)Z conditions
Unused since TestNZ+NZCVSelect accomplishes the same and good riddance. Might
bring them back later for tbz/tbnz, but certainly not in this Selectful form.

(I added them when I thought we were going to RA the flags. With the more
effective static approach, we don't need this for that.)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 472d143021 OpcodeDispatcher: Use TestNZ directly for SelectBit
Lets us drop the bitwise select variants, this was the last use.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 157f95b08f IR: Extend TestNZ to two sources
Unlock the power of AND.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 6bf7ab0778 IR: Remove complex branches
Performance footgun and now unused. Don't bring it back.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 0de958be2a IR: Remove Abs
Now unused. If we bring it back, it should be brought back as CSSC only. On
non-CSSC platforms, an explicit cmp + predicated neg can be better.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig ef544fecf2 OpcodeDispatcher: Use predicated neg for x87 fild
Saves an instruction on non-CSSC platforms by deleting a redundant cmp.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 5eea68d6c6 IR: Support predicated Neg
i.e. cneg. will be used for x87 hell op

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 5471367db1 OpcodeDispatcher: Optimize branches
For native cases. Big perf win.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig b8265b1067 OpcodeDispatcher: Rework SelectCC for branches
Separate out the NZCV bits from the more complex stuff so we can specially
optimize the branches.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Alyssa Rosenzweig 099c683a5a IR: Add FromNZCV mode to CondJump
In this mode, rather than the branch comparing its arguments and then jumping
based on the result, the branch simply jumps by the native comparison based on
the NZCV value. This allows us to map x86 branches to arm64 branches 1:1.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-12 17:32:24 -04:00
Ryan Houdek 0357bb23e8 Merge pull request #3265 from Sonicadvance1/gdb_unix_socket
GdbServer: Switch over to a unix domain socket
2023-11-11 13:20:40 -08:00
Ryan Houdek c7193b52fb GdbServer: Switch over to a unix domain socket
Resolving the issue that we can only ever have one gdbserver process
running consuming port 8086 (even if the port number is cute).

Doesn't give us anything yet but in the future will allow us to have
whole process trees running gdbservers that we can attach to.
2023-11-11 13:14:09 -08:00
Alyssa Rosenzweig ec14a65e23 OpcodeDispatcher: optimize LOOP invert
need to reorder since the select clobbers nzcv. before/after diff on the loopne
unit test:

> 4308: [INFO] cset w20, ne
40c41
< 4308: [INFO] mrs x20, nzcv
---
> 4308: [INFO] mrs x21, nzcv
42,49c43,48
< 4308: [INFO] cset x21, ne
< 4308: [INFO] ubfx w22, w20, #30, #1
< 4308: [INFO] eor x22, x22, #0x1
< 4308: [INFO] and x21, x21, x22
< 4308: [INFO] msr nzcv, x20
< 4308: [INFO] cbnz x21, #+0x8 (addr 0xfffed66f8094)
< 4308: [INFO] b #+0x1c (addr 0xfffed66f80ac)
< 4308: [INFO] ldr x0, pc+8 (addr 0xfffed66f809c)
---
> 4308: [INFO] cset x22, ne
> 4308: [INFO] and x20, x22, x20
> 4308: [INFO] msr nzcv, x21
> 4308: [INFO] cbnz x20, #+0x8 (addr 0xfffec94e8090)
> 4308: [INFO] b #+0x1c (addr 0xfffec94e80a8)
> 4308: [INFO] ldr x0, pc+8 (addr 0xfffec94e8098)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 21:03:58 -04:00
Alyssa Rosenzweig 0d70c6a0d0 OpcodeDispatcher: Use cset for getting nzcv flags
Usually better in practice... some rotates are slightly regressed by this but
they were already terrible.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 21:03:58 -04:00
Alyssa Rosenzweig bfa069c4d5 OpcodeDispatcher: Dirty NZCV on new blocks
This worked only by accident before.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 21:03:58 -04:00
Alyssa Rosenzweig 61bdf64e15 OpcodeDispatcher: Cleanup PF select
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 21:03:58 -04:00
Alyssa Rosenzweig 482b35c283 OpcodeDispatcher: Optimize JA/JNA selects
Chain two csel instructions together, which should be optimal.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 10:46:59 -04:00
Alyssa Rosenzweig b74d886017 OpcodeDispatcher: Cleanup selectcc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 10:46:59 -04:00
Alyssa Rosenzweig 1667abad7e OpcodeDispatcher: Don't fold flags in SelectCC
Not beneficial in the new approach.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 10:46:59 -04:00
Alyssa Rosenzweig b187a853e7 OpcodeDispatcher: Use NZCVSelect for SelectCC
Massively better codegen.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 10:46:58 -04:00
Alyssa Rosenzweig 228c7d142e IR: Allow inline 0 on NZCVSelect
via the 0 reg. We really need a more generalized approach to taking advantage of
wzr, but this optimizes the special case I care about for seta (saving a move to
make the impl optimal).

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 09:42:26 -04:00
Alyssa Rosenzweig 041199644c IR: Add NZCVSelect op
This will replace Select soon, as it lets us take advantage of
NZCV-generating instructions and it doesn't clobber NZCV.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-10 09:04:42 -04:00
Alyssa Rosenzweig 3767f3633d Merge pull request #3263 from alyssarosenzweig/opt/not-garbage
OpcodeDispatcher: Make "not" not garbage
2023-11-09 19:58:02 -04:00
Ryan Houdek af3253947e Merge pull request #3162 from alyssarosenzweig/opt/nzcv-native
Keep guest SF/ZF/CF/OF flags resident in host NZCV
2023-11-09 15:10:45 -08:00
Ryan Houdek efc5eb2933 Merge pull request #3250 from Sonicadvance1/gdbserver_frontend_move
FEXLoader: Wire up gdbserver in the frontend
2023-11-09 14:48:59 -08:00
Ryan Houdek b4eeb96375 Merge pull request #3261 from Sonicadvance1/tuning_options
FEX: Only pass CPU tunables to FEXCore and FEXLoader
2023-11-09 14:48:42 -08:00
Alyssa Rosenzweig c9832e3d34 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 15:22:25 -04:00
Alyssa Rosenzweig da3e3fc7a3 OpcodeDispatcher: Optimize some selects
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 15:21:10 -04:00
Alyssa Rosenzweig b5c83f0628 JIT: Optimize 8/16-bit TestNZ
This is cursed. I blame the darling.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 14:50:36 -04:00
Alyssa Rosenzweig 03087a55ba InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 12:02:20 -04:00
Alyssa Rosenzweig 1ce3c16b30 OpcodeDispatcher: Make "not" not garbage
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 12:02:20 -04:00
Alyssa Rosenzweig bf702850a9 InstCountCI: Add not bh case
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 11:32:45 -04:00
Alyssa Rosenzweig 584c4cc05e OpcodeDispatcher: Mask with rmif sometimes
For CF/OF calculation, this saves an instruction on flagm platforms.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 10:05:51 -04:00
Alyssa Rosenzweig 279afd88bb OpcodeDispatcher: Generalize rmif trick
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 10:05:51 -04:00
Alyssa Rosenzweig c0a6d82025 OpcodeDispatcher: Use rmif for NZCV inserts
Optimizes piles of s/w flag generation on flagm.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 3a03e1c93c OpcodeDispatcher: rework InsertNZCV
in prep for rmif.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig bdaa70405f OpcodeDispatcher: simplify flag control ops
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 1281145982 OpcodeDispatcher: Optimize popf
mostly for easier debugging tbh

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 87cac09477 OpcodeDispatcher: optimize cmc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 5336129b58 IR: Optimize sub/sbb
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 72fc2b522d IR: Optimize add/adc
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig b6f6c84790 IR: Optimize tests
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 11e9be13b1 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig afdb8753ba IR: Remove some implicit flag clobbers
Do it explicitly for sve-256 and punt on optimizing, so we avoid regressing code
gen otherwise.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig f6a2e6739d IR: VCMPEQ doesn't clobber nzcv
cmeq not cmpeq!

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig d6569d510d Arm64: Keep host flags resident in NZCV
Rather than the context. Effectively a static register allocation scheme for
flags. This will let us optimize out a LOT of flag handling code, keeping things
in NZCV rather than needing to copy between NZCV and memory all the time.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 04e4993d9b OpcodeDispatcher: Add a kludge to save NZCV less
Some opcodes only clobber NZCV under certain circumstances, we don't yet have
a good way of encoding that. In the mean time this hot fixes some would-be
instcountci regressions.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 783e09d67d ConstProp: Remove select code motion
Problematic in the new approach and not sure what it's trying to accomplish tbh.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 314f478225 ConstProp: remove select+branch fusion
Not beneficial in the new approach to flags.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig c1dbc28aa2 OpcodeDispatcher: Implement SaveNZCV
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig 8f7e393ffb Arm64: Don't clobber NZCV in CondJump
Again we need to handle this one specially because the dispatcher can't insert
restore code after the branch. It should be optimized in the near future, don't
worry.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig b3055523b4 IR: Switch to dedicated NZCV load/store
Semantics differ markedly from the non-NZCV flags, splitting this out makes it a
lot easier to do things correctly imho. Gets the dest/src size correct
(important for spilling), as well as makes our existing opt passes skip this
which is needed for correctness at the moment anyway.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Alyssa Rosenzweig cf6b21564c InstCountCI: disable flagm explicitly more
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-09 09:40:51 -04:00
Mai 996a4c023c Merge pull request #3260 from Sonicadvance1/ender_lilies_nzcv_unittest
unittests/ASM: Adds unittest found from Ender Lilies that crashed with NZCV
2023-11-08 20:24:09 +01:00
Ryan Houdek 0dcbdcc0e2 FEX: Only pass CPU tunables to FEXCore and FEXLoader
This fixes an issue where CPU tunables were ending up in the thunk
generator which means if your CPU doesn't support all the features on
the *Builder* then it would crash with SIGILL. This was happening with
Canonical's runners because they typically only support ARMv8.2 but we
are compiling packages to run on ARMv8.4 devices.

cc: FEX-2311.1
2023-11-08 05:50:33 -08:00
Ryan Houdek 5bdd422db6 unittests/ASM: Adds unittest found from Ender Lilies that crashed with NZCV
SHA instructions are very large right now and cause register spilling
due to their codegen. Ender Lilies has a really large block in a
function called `sha1_block_data_order` that was causing FEX to spill
NZCV flags incorrectly. The assumption which held true before NZCV
optimizations were a thing was that all flags were either 1-bit in an
8-bit container, or just 8-bit (x87 TOP flag).

NZCV host flags broke this assumption by making its flags 32-bit which
ended up breaking when encounting spilling situations.
2023-11-08 04:52:22 -08:00
Alyssa Rosenzweig bf147f47b5 Merge pull request #3258 from Sonicadvance1/remove_warnings_16
Arm64Emitter: Fixes warning
2023-11-08 07:05:26 -04:00
Alyssa Rosenzweig 3f1f7faf34 Merge pull request #3257 from alyssarosenzweig/helper/derive-op
Add helper for deriving ops by opcode
2023-11-08 06:53:45 -04:00
Ryan Houdek 1fc6725826 Arm64Emitter: Fixes warning 2023-11-08 01:27:27 -08:00
Ryan Houdek fa8c35feba Docs: Update for release FEX-2311 2023-11-07 09:42:16 -08:00
Alyssa Rosenzweig 73958b9163 OpcodeDispatcher: Use DeriveOp
Replace every instance of the Op overwrite pattern, and ban that anti-pattern
from the codebase in the future. This will prevent piles of NZCV related
regressions.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-07 12:05:00 -04:00
Alyssa Rosenzweig 9b81a83894 OpcodeDispatcher: Add DeriveOp helper
The "create op with wrong opcode, then change the opcode" pattern is REALLY
dangerous. This does not address that. But when we start doing NZCV trickery, it
will get /more/ dangerous, and so it's time to add a helper and make the
convenient thing the safe(r) thing. This helper correctly saves NZCV /before/
the instruction like the real builders would. It also provides a spot for future
safety asserts if someone is motivated.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-07 12:05:00 -04:00
Alyssa Rosenzweig 65b7d4007e Merge pull request #3256 from Sonicadvance1/unittest_for_3254
unittests/ASM: Adds unittest for implicit flag clobber for #3254
2023-11-07 10:02:07 -04:00
Ryan Houdek 2c0444e846 unittests/ASM: Adds unittest for implicit flag clobber for #3254
It looks like currently FEX has a bug around implicit flag clobbering
with this pull request where an IR operation that implicitly clobbers
flags isn't correctly saving the NZCV flags before doing the operation.

Adds a unit test that specifically captures this issue. RAX will be 1 or
0 depending on if the flags are clobbered incorrectly or not.
2023-11-07 02:04:11 -08:00
Ryan Houdek b92e716d0c Merge pull request #3255 from Sonicadvance1/unittest_flags_signals
FEXLinuxTests: Adds a unittest for eflags and signals around a inlined syscall
2023-11-06 05:43:44 -08:00
Ryan Houdek a7a1365cf7 FEXLinuxTests: Adds a unittest for eflags and signals around a inlined syscall
When attempting to debug #3162 I had noticed spurious behaviour around
what I assumed to be eflags getting corrupt around inlined syscalls.
This turned out to be a red herring but to ensure we are still testing
this, create a fully fleshed out unit test.

This test ensures a couple of things.
1) A flag that is set or unset before a syscall doesn't have its data
   corrupt
2) An inline syscall doesn't corrupt the eflags, checking the eflag
   result after returning from the syscall.
3) A signal occuring while in an inline syscall returns the correct
   eflags information in the signal handler information

This test gets accomplished by setting or unsetting a particular flag
and then calling the futex syscall in a way that is guaranteed to be
inlined and also wait forever. Then the parent thread will signal with a
SIGTERM and read back the signal information. It does this multiple
times for each flag we care about.
2023-11-06 05:33:03 -08:00
Alyssa Rosenzweig 8ee5b5cf50 Merge pull request #3253 from Sonicadvance1/fix_double_munmap
JITArm64: Fixes double munmap issue that was causing crashes
2023-11-06 07:57:01 -04:00
Alyssa Rosenzweig b45023bedf Merge pull request #3251 from Sonicadvance1/fexlinuxtests_use_intel
FEXLinuxTests: Compile tests with masm=intel
2023-11-06 07:56:29 -04:00
Alyssa Rosenzweig 3702e513f5 Merge pull request #3252 from Sonicadvance1/fix_fillstaticregisters
ARMEmitter: Fix GPR fill mask in `FillStaticRegs`
2023-11-06 07:50:58 -04:00
Ryan Houdek 829384e488 JITArm64: Fixes double munmap issue that was causing crashes
While tracking issues in #3162, I had encountered a random crash that I
started hunting. It was very quickly apparent that this crash was
unrelated to that PR. I just happened to be running a unittest that was
creating and tearing down a bunch of threads that exacerbated the
problem.

See as follows with the strace output:
```
[pid 269497] munmap(0x7fffde1ff000, 16777216) = 0
[pid 269497] munmap(0x7fffde1ff000, 16777216 <unfinished ...>
[pid 268982] mmap(NULL, 16777216, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fffde1ff000
[pid 269497] <... munmap resumed>)      = 0
```

One thread is freeing some memory with munmap, another one then does a mmap and gets the same address back.
Nothing too crazy at initial glance, but taking a closelier look, we can
see that there are two strange oddities:
1) We are double unmapping the same address range through munmap
2) The second munmap is interrupted and returns AFTER the mmap.

This has the unfortunate side-effect that the mmap that just returned
the same address has actually just been unmapped! This was resulting in
spurious crashes around thread creation that was SUPER hard to nail
down.

The problem comes down to how code buffer objects are managed, in
particular how the Arm64Emitter and Dispatcher handled its buffers.

Arm64Emitter is inherited by two classes; Dispatcher, and Arm64JITCore.
On class destruction the emitter would free its internal tracking
buffer. Additionally on destruction, the Arm64JITCore would walk through
all of its CodeBuffers and free them. The problem ends up being that in
the Arm64JITCore, it would free its code buffers which also ended up
being the current active buffer bound to the Arm64Emitter. Thus causing
the Arm64Emitter to come back around and try to free the same buffer
again.

This is a double-free problem! and was only visible on thread exiting!
Can't track double frees with mmap and munmap with current tooling!

This problem typically didn't occur because of how fast the destruction
usually takes and jemalloc inbetween also typically means the problem
doesn't occur. Initially thinking this was a threaded pool allocator bug
because typically the new allocation would end up in there once a new
thread was spinning up.

Now we change behaviour, Arm64Emitter doesn't do any buffer management
itself, instead just passing an initial buffer on to its internal buffer
tracking if given one up front.

This leaves the Dispatcher and the Arm64JITCore to do their buffer
management and ensuring there is no double free.

The day is saved!
2023-11-05 19:18:40 -08:00
Ryan Houdek ed23fbe932 InstCountCI: Update for SRA spill changes. 2023-11-05 02:59:19 -08:00
Ryan Houdek 0af0427efd ARMEmitter: Fix GPR fill mask in FillStaticRegs
This mask was being used incorrectly, it's a GPR spill mask for host
GPRs not an index in to the SRA array. Search the array of SRA registers
for the first one in the mask first to use as a temporary.

Fixes an issue with 32-bit inline syscalls where the first register
being spilled was r8, which was beyond the size of SRA registers on
32-bit processes. This would cause FEX to read the value just after
x32::SRA which is x32::RA. This would mean it would use r20 as a
temporary, corrupting the register in the process.

I noticed this while poking at #3162, but also when I was looking at a
memory buffer ownership problem.
2023-11-05 02:54:48 -08:00
Ryan Houdek a499272d81 FEXLinuxTests: Compile tests with masm=intel
ATT is maddening and a unittest I'm writing was going to be a nightmare.
2023-11-04 22:00:45 -07:00
Ryan Houdek 5dee921300 FEXLoader: Wire up gdbserver in the frontend
Requires #3249 to be merged first

Library alerting has been disabled for now, and storing IR while
gdbserver is running is removed.

Otherwise no functional change.
2023-11-03 20:23:51 -07:00
Ryan Houdek c0dcf8925a Move gdbserver to frontend 2023-11-03 20:23:51 -07:00
Ryan Houdek e91c5ff906 Merge pull request #3249 from Sonicadvance1/gdbserver_frontend_prep
GDBServer: Preparation work to get this moved to the frontend
2023-11-03 20:21:28 -07:00
Ryan Houdek 190f7c27e0 Merge pull request #3243 from Sonicadvance1/loadfile_unsized
FEXCore/FileLoading: Updates helper to load file that is backed by memory
2023-11-03 20:21:12 -07:00
Ryan Houdek b15f0b5d36 FEXCore/FileLoading: Updates helper to load file that is backed by memory
When attempting to read files that aren't backed by a filesystem then
our current read file helpers fail since they query the file size
upfront.

Change the helper so that it doesn't query the size and just reads the file if it
can be opened. This lets us read `/proc/self/maps` using helpers.
2023-11-03 07:01:39 -07:00
Ryan Houdek e2c65189ff GDBServer: Preparation work to get this moved to the frontend
GDBServer is inherently OS specific which is why all this code is
removed when compiling for mingw/win32. This should get moved to the
frontend before we start landing more work to clean this interface up.

Not really any functional change.

Changes:

FEXCore/Context: Adds new public interfaces, these were previously
private.
- WaitForIdle
   - If `Pause` was called or the process is shutting down then this
     will wait until all threads have paused or exited.
- WaitForThreadsToRun
   - If `Pause` was previously called and then `Run` was called to get
     them running again, this waits until all the threads have come out
     of idle to avoid races.
- GetThreads
   - Returns the `InternalThreadData` for all the current threads.
   - GDBServer needs to know all the internal thread data state when the
     threads are paused which is what this gives it.

GDBServer:
- Removes usages of internal data structures where possible.
   - This gets it clean enough that moving it out of FEXCore is now
     possible.
2023-11-02 20:11:01 -07:00
Ryan Houdek 03f63f99a8 FEXCore: Moves StringUtils to FEXCore headers
Once gdbserver gets moved to the frontend this will need to be in the
includes.
2023-11-02 20:09:12 -07:00
Ryan Houdek e305a9a0d5 Merge pull request #3248 from neobrain/fix_thunks_async_callback_error
Thunks: Print error if guest-provided callbacks are called asynchronously
2023-11-02 12:03:22 -07:00
Tony Wasserka 3a90dbbb35 Thunks: Print error if guest-provided callbacks are called asynchronously from the host 2023-11-02 19:50:58 +01:00
Ryan Houdek 5103f2d92b Merge pull request #3247 from alyssarosenzweig/refactor/nzcv-prereq
Preparatory patches for nzcv
2023-11-01 14:07:14 -07:00
Alyssa Rosenzweig d4a6b031ea Merge pull request #3245 from Sonicadvance1/remove_gdbpausecheck
FEXCore: Removes gdb pause check handler
2023-11-01 15:46:40 -04:00
Alyssa Rosenzweig 319cf4bf3d Arm64: Preserve flags in ExitFunction
Mostly harmless (except for fusion on cortexes), prepares us for nzcv work.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-01 15:44:36 -04:00
Alyssa Rosenzweig d75c0f2c50 IR: Add missing flag clobbers
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-01 15:44:36 -04:00
Alyssa Rosenzweig a586d3823d IR: VFCMPEQ does not clobber flag
Oversight.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-01 15:44:36 -04:00
Alyssa Rosenzweig 5522c6db9c OpcodeDispatcher: Use jump wrappers
Mostly automated replacement + renaming for build fixing.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-01 15:44:36 -04:00
Alyssa Rosenzweig 367e1658ad OpcodeDispatcher: Add jump wrappers
These should always be used in the dispatcher rather than the raw jumps they
translate to, as they ensure that flags are flushed. Eliminates a class of bugs
that will become a lot easier to hit with the new nzcv work.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-01 15:44:36 -04:00
Alyssa Rosenzweig bbad06f81a ArchHelpers: Add cfinv()
FlagM goodness.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-11-01 15:44:35 -04:00
Mai 9612b2fe4b Merge pull request #3240 from Sonicadvance1/optimize_palignr_zero
OpcodeDispatcher: Optimize palignr with zero immediate
2023-11-01 05:55:52 +01:00
Mai ef5503f0b7 Merge pull request #3239 from Sonicadvance1/optimize_blendw
OpcodeDispatcher: Optimize pblendw
2023-11-01 05:54:31 +01:00
Ryan Houdek 26bf67ca76 InstCountCI: Update for pblendw 2023-10-31 21:20:05 -07:00
Ryan Houdek e5df636efd OpcodeDispatcher: Optimize pblendw
Requires #3238 to be merged first since this uses the tbx IR operation.

Worst case is now a three instruction sequence of ldr+ldr+tbx.
Some operations are special-cased, which definitely doesn't cover all
possible cases we could use without tbx, but as a worst case improvement
this is a significant improvement.
2023-10-31 21:18:02 -07:00
Ryan Houdek f34f4a0227 unittests/ASM: Extend pblendw test for all swizzles
We can't sanely test all 256 swizzle masks, so walk the array of random
data for each swizzle and CRC the results each step along the way.
This will give us a final result rather than each individual step.
2023-10-31 21:16:19 -07:00
Mai f45722d2cd Merge pull request #3238 from Sonicadvance1/optimize_blendps
OpcodeDispatcher: Optimize blendps
2023-11-01 04:32:33 +01:00
Ryan Houdek db7ef0e4b0 InstCountCI: Update for blendps
All sixteen swizzles are now optimal.
2023-10-31 20:14:56 -07:00
Ryan Houdek de10cbad98 OpcodeDispatcher: Optimize blendps
A bunch of blendps swizzles weren't optimal. This optimizes all swizzles
to be optimal.

Two instructions can be more optimal without a tbx but the rest required
tbx to be optimal since they don't match ARM's swizzle mechanics.
2023-10-31 20:06:45 -07:00
Ryan Houdek e4d9c264d8 IR: Adds support for tbx 2023-10-31 20:06:45 -07:00
Ryan Houdek 97b3efa90a unittests/ASM: Update blendps to test all swizzles
There are sixteen swizzles so it can fit in a single test.
2023-10-31 20:06:45 -07:00
Mai 18065199e3 Merge pull request #3242 from Sonicadvance1/gdbserver_threadnames
GdbServer: Fixes returning thread names
2023-11-01 04:02:14 +01:00
Mai 77d92872bc Merge pull request #3212 from Sonicadvance1/dpp_opt
OpcodeDispatcher: Optimize 128-bit DPPS and DPPD
2023-11-01 04:01:05 +01:00
Ryan Houdek 460f13be71 FEXCore: Removes gdb pause check handler
gdbserver is currently entirely broken so this doesn't change behaviour.
The gdb pause check that we originally had an excessive amount of
overhead.
Instead use the pending interrupt fault check that was wired
up for wine.
This makes the check very lightweight and makes it more reasonable to
implement a way to have gdbserver support attaching to a process.
2023-10-31 18:40:00 -07:00
Ryan Houdek 47c9463217 GdbServer: Fixes returning thread names
gdb gets angry if we return text with `<No Name>` in an xml text field.
Instead only return a name if we have one and gdb will take care of the
rest.

Additionally change the formatting of the return packet, it doesn't need
the xml version header.
2023-10-29 17:16:46 -07:00
Ryan Houdek 15c825f362 Merge pull request #3241 from neobrain/refactor_thunks_drop_async
Thunks/xcb: Drop unused and incomplete support for asynchronous callbacks
2023-10-29 13:27:29 -07:00
Alyssa Rosenzweig bbd20b47ba Merge pull request #3236 from Sonicadvance1/fix_strenum
Config: Fixes string enum parser with multiple arguments
2023-10-29 07:32:57 -04:00
Alyssa Rosenzweig 5b70209728 Merge pull request #3233 from Sonicadvance1/print_vixl
JIT: Implements Print support for vixl sim
2023-10-29 07:32:32 -04:00
Alyssa Rosenzweig a287f2a189 Merge pull request #3237 from Sonicadvance1/round_classification
InstCountCI: Update rounds{s,d} classification
2023-10-29 07:31:22 -04:00
Tony Wasserka 3a240e3b61 Thunks/xcb: Drop unused and incomplete support for asynchronous callbacks 2023-10-29 11:05:24 +01:00
Ryan Houdek de1e593ec2 InstCountCI: Update for palignr zero immediate 2023-10-27 15:11:07 -07:00
Ryan Houdek 13cd8b33a2 OpcodeDispatcher: Optimize palignr with zero immediate
These turns in to moves
2023-10-27 15:09:31 -07:00
Ryan Houdek 0f26bc20a3 unittests/ASM: Adds palignr tests for zero immediate
These effectively turn in to moves.
2023-10-27 15:07:52 -07:00
Ryan Houdek eacab3cc22 InstCountCI: Update rounds{s,d} classification
This is optimal without AFP. We already test these with AFP in a
different file.
2023-10-27 12:38:02 -07:00
Ryan Houdek 09e3371a0d Config: Fixes string enum parser with multiple arguments
Messed up when originally implementing this, substr's second argument is
requested substring length, not the ending position.

Noticed this while trying to parse multiple FEX_HOSTFEATURES options.
2023-10-27 12:04:04 -07:00
Alyssa Rosenzweig ff3f7345b6 Merge pull request #3235 from Sonicadvance1/cpu_names
CPUID: Adds some missing cpu core names
2023-10-27 08:00:43 -04:00
Alyssa Rosenzweig 8181e53727 Merge pull request #3234 from Sonicadvance1/minor_bfxil
Arm64: Minor optimization to bfxil and bfi
2023-10-27 08:00:28 -04:00
Alyssa Rosenzweig 5431aa5a28 Merge pull request #3232 from Sonicadvance1/assert_code
IR: Print assert code for IR EmitValidation
2023-10-27 07:58:51 -04:00
Ryan Houdek 1a293cc542 CPUID: Adds some missing cpu core names
EZ-PZ
2023-10-26 21:03:32 -07:00
Ryan Houdek b1e78934ad InstCountCI: Improvement for bfi/bfxil 2023-10-26 20:26:48 -07:00
Ryan Houdek 61f22911c7 Arm64: Minor optimization to bfxil and bfi
When the destination doesn't alias the source, we can remove a final mov
from both of these operations.

Does some minor code improvement.
2023-10-26 20:25:44 -07:00
Ryan Houdek fe8778bb96 JIT: Implements Print support for vixl sim
Everytime I want to quickly output a value for testing I tend to use
Print which didn't work under the simulator.
Give this a quick fix to wire up the jump to the vixl sim.
2023-10-26 18:34:47 -07:00
Ryan Houdek 14e5ea1e22 Merge pull request #3230 from Sonicadvance1/optimize_atomic_fetch
IR: Optimize unused result atomic fetch mop to just atomic mop
2023-10-26 15:57:20 -07:00
Ryan Houdek 74f1205f33 IR: Print assert code for IR EmitValidation
Currently we don't get why an IR emit failed in the assert message. Put
the code in to the message so it is easier to see.
This also resolved the issue that when in RelWithDebInfo the assert line
would typically be the end of the IR emission function, so you couldn't
see which assert actually triggered. Now since the message is printed
this is easier

Before:
```
[ASSERT]
```

After:
```
[ASSERT] Size == FEXCore::IR::OpSize::i32Bit || Size == FEXCore::IR::OpSize::i64Bit
```

This is a lot easier and better data than what #3227 proposed.
2023-10-26 15:51:56 -07:00
Ryan Houdek 4045bfd187 Merge pull request #3229 from Sonicadvance1/instcountci_3dnow
OpcodeDispatcher: Optimize a few 3DNow! operations
2023-10-26 15:42:50 -07:00
Ryan Houdek 9db93a43dd IR: Adds comments that atomic op layout must match
If the fetch and nonfetch versions mismatched then the DCE optimization
which changes the IR operation would break handily.

Add a comment as a reminder so if anyone touches this they will
understand.
2023-10-26 15:40:55 -07:00
Ryan Houdek a379d50729 Merge pull request #3226 from neobrain/fix_thunks_incomplete_but_compatible
Thunks: Skip data layout analysis for types that are always assumed compatible
2023-10-26 13:44:31 -07:00
Alyssa Rosenzweig f5822f83b0 Merge pull request #3231 from Sonicadvance1/alu_ops_more_garbage
OpcodeDispatcher: Allow garbage in upper bits for more ALU ops
2023-10-26 06:23:14 -04:00
Ryan Houdek 5028434292 IR: Optimize unused result atomic fetch mop to just atomic mop
When the result of an atomic fetch operation is unused then we can
safely convert it to a non-fetching version of the operation.

This happens hundreds of times per process as far as I can tell. No idea
if this actually helps any hardware, but theoretically it can allow CPUs
to not stall waiting for writeback of the atomic operation.
Couldn't detect any performance improvements in the various little
things I was poking at least. Very trivial to support so add it.

Unaligned variants are already handled in our unaligned fault handler
since the only difference is the acquire semantic is dropped and the
destination register is the zero register.
2023-10-25 21:00:04 -07:00
Ryan Houdek 8f0461cac8 IR: Adds support for non-fetch Atomci CLR and NEG 2023-10-25 21:00:04 -07:00
Ryan Houdek 0b4cb23411 InstCountCI: Update for more garbage data 2023-10-25 20:46:31 -07:00
Ryan Houdek bab96b9441 OpcodeDispatcher: Allow garbage in upper bits for more ALU ops
Secondary ALU operations were missed and when the operation is 4-bytes
in size we can also allow garbage upper bits since the JIT will emit a
32-bit operation for this instruction which is safe.

Optimizes some bad codegen around 32-bit ALU operations.
2023-10-25 20:44:36 -07:00
Ryan Houdek 11e2f14185 InstCountCI: Update for 3DNow! changes 2023-10-25 19:30:13 -07:00
Ryan Houdek 6177290e9d OpcodeDispatcher: Optimize a few 3DNow! operations
Instead of using VInsElement in pi2fw and pf2iw, just use uzp1 to ensure
we don't unintentionally add to RA pressure.

Additionally we can generate the constant needed for pmulhrw directly
using the movi instruction. Converts two instructions in to one.

Under FEX's current constraints this makes all 3DNow! instructions
optimal.
2023-10-25 19:27:54 -07:00
Ryan Houdek 20a54913bd IR: Support shifted imms in VectorImm
We will want to use this in some cases.
2023-10-25 19:27:00 -07:00
Tony Wasserka dd9ed89a7a Thunks/X11: Drop unneeded type annotation 2023-10-25 20:38:57 +02:00
Tony Wasserka a4e1e0a1fb Thunks/gen: Skip data layout analysis for types that are always assumed compatible 2023-10-25 19:35:02 +02:00
Tony Wasserka 5e9f69001d Thunks/gen: Clarify pointer parameter handling
One of the subconditions was always true, so it can safely be removed.
2023-10-25 19:35:02 +02:00
Ryan Houdek 39c5ab1c81 Merge pull request #3225 from neobrain/fix_32bit_funcptrs
Thunks: Fix function pointer support on 32-bit
2023-10-25 07:07:18 -07:00
Ryan Houdek 5bf790324c Merge pull request #3224 from neobrain/feature_thunk_strict_everywhere
Thunks: Annotate pointer parameters throughout all thunked libraries
2023-10-25 07:00:42 -07:00
Tony Wasserka adcdb32d49 Thunks: Fix function pointer support on 32-bit 2023-10-25 14:37:22 +02:00
Tony Wasserka f264578f12 Thunks: Unconditionally enable strict processing mode 2023-10-25 12:39:57 +02:00
Tony Wasserka 7149da387a Thunks: Annotate pointer parameters throughout all thunked libraries 2023-10-25 12:39:57 +02:00
Alyssa Rosenzweig 0f3d14e7c0 Merge pull request #3223 from Sonicadvance1/testharness_page_size
TestHarnessRunner: Don't hardcode stack allocation to 4096 bytes
2023-10-24 10:52:35 -04:00
Ryan Houdek aad5080224 TestHarnessRunner: Don't hardcode stack allocation to 4096 bytes
Just allocate a single page that we query at runtime.
2023-10-24 07:36:13 -07:00
Ryan Houdek c77a3d673c Merge pull request #3222 from Sonicadvance1/nzcv_opt_bug
unittests: Adds test for bug from #3162
2023-10-23 18:19:16 -07:00
Ryan Houdek 0ff2e6e1e3 unittests: Adds test for bug from #3162
This PR has a bug around flags calculation and REP LODS{B,W,D,Q}.
This currently passes on main but fails on #3162.

Bug only occurs in 32-bit instead of 64-bit with the same test. Should
help diagnose the bugs in #3162.
2023-10-23 16:55:56 -07:00
Ryan Houdek 8538f5bac4 Merge pull request #3221 from Sonicadvance1/instcountci_missing_insts
InstCountCI: Adds two missing variants of movd/movq
2023-10-23 15:26:58 -07:00
Ryan Houdek a305baf6e5 InstCountCI: Fixes some mislabeled instructions
These are optimal according to our standards.
2023-10-23 15:15:03 -07:00
Ryan Houdek 807619aa02 InstCountCI: Adds two missing variants of movd/movq
We only had the move to memory destination version, ensure we test to
GPR as well.
2023-10-23 15:01:18 -07:00
Ryan Houdek 6db2125b41 Merge pull request #3220 from Sonicadvance1/override_flagm
InstCountCI: Support disabling flagm extensions
2023-10-23 14:45:45 -07:00
Ryan Houdek 9f6d80fe5d InstCountCI: Duplicate tests that change behaviour based on flagm
Necessary for #3162 to have consistent behaviour in CI
2023-10-23 14:03:19 -07:00
Ryan Houdek 423ce12001 InstCountCI: Disable flagm and Flagm2 on tests
Most of these will get duplicated in the next commit
2023-10-23 14:02:50 -07:00
Ryan Houdek 4edd72fc33 InstCountCI: Support disabling flagm extensions
This is necessary so #3162 can give consistent results
2023-10-23 14:02:24 -07:00
Ryan Houdek 978f607dd9 Merge pull request #3177 from neobrain/feature_thunk_pointer_annotations
Thunks: Add new pointer annotations to assist data layout analysis
2023-10-23 13:00:27 -07:00
Ryan Houdek 9ba78c9771 Merge pull request #3219 from Sonicadvance1/enable_vixlsim_instcountci
github: Enables Vixl simulator on x86 host for instcountci
2023-10-23 12:49:33 -07:00
Ryan Houdek e2144345c0 github: Enables Vixl simulator on x86 host for instcountci
Otherwise features get filled out weirdly.
2023-10-23 12:42:40 -07:00
Ryan Houdek 63e4c3682d Merge pull request #3218 from Sonicadvance1/opt_df
OpcodeDispatcher: Optimize DF pointer offset calculation
2023-10-23 11:41:22 -07:00
Ryan Houdek 2956e84ead InstCountCI: Update for constant prop improvement 2023-10-23 10:39:29 -07:00
Ryan Houdek 4466c50c2b ConstProp: Optimize SubShift and Add with negative
When SubShift (LSL) occurs with both sources constant then optimize away
the calculation.

Additionally if add is found to have one immediate constant where the
inverse of the constant fits in to ImmAddSub range, then invert the
constant and change it in to a sub.

This optimizes the cases when direction flag is known upfront in an
instruction.
2023-10-23 10:36:33 -07:00
Ryan Houdek dd5ca1d349 InstCountCI: Update for DF pointer optimization. 2023-10-23 10:14:51 -07:00
Ryan Houdek 95c756b466 OpcodeDispatcher: Optimize DF pointer offset calculation
Previously this moved two constant, did a compare and a csel. Four
instructions in total. It also corrupts NZCV which we want to use for
other things.

This new codegen emits one constant and one subtract instruction, two
instructions total and doesn't touch NZCV.

More optimal!
2023-10-23 09:27:41 -07:00
Ryan Houdek 99465faf63 IR: Implements support for subtract with shifted register
Will be used soon.
2023-10-23 09:27:41 -07:00
Ryan Houdek e018917f76 Merge pull request #3216 from alyssarosenzweig/opt/nzcv-infra
Prep commits for NZCV modelling
2023-10-23 07:39:47 -07:00
Ryan Houdek a261d9909e Merge pull request #3214 from Sonicadvance1/fix_bug
FEXCore: Fixes bug in vector `ZextAndMaskingElimination` pass
2023-10-23 07:29:43 -07:00
Alyssa Rosenzweig 6de8bc6848 IR: Annotate instructions with implicit flag clobber
Audit the code base and mark any instruction that implicitly clobbers flags so
it can get special handling in the dispatcher to spill NZCV ahead of emitting.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 10:21:47 -04:00
Alyssa Rosenzweig d87155e4ee IR: Add infrastructure for modelling flag clobbers
Lots of instructions clobber NZCV inadvertently but are not intended to write to
the host flags from the IR point-of-view. As an example, Abs logically has no
side effects but physically clobbers NZCV due to its cmp/csneg impl on non-CSSC
hw. Add infrastructure to model this in the IR so we can deal with it when we
start using NZCV for things.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 10:21:47 -04:00
Alyssa Rosenzweig 7484cacaf9 InstCountCI: Update for VInsertElement change
Only SVE256 codepath affected.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 10:21:47 -04:00
Alyssa Rosenzweig 42259974c4 Arm64: Preserve NZCV in VInsertElement
So we don't need to mark VInsertElement as implicit clobber in the common case.
Only afects sve256 which doesn't exist yet.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 10:21:34 -04:00
Alyssa Rosenzweig e455996dbd OpcodeDispatcher: Remove silly shift branching
The flag generation code does this internally and more efficiently.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 10:16:38 -04:00
Alyssa Rosenzweig b5dd1d05e9 Dispatcher: Preserve NZCV
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 10:16:38 -04:00
Alyssa Rosenzweig bbaf70da15 Dispatcher: Yeet pointless subs
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 10:16:29 -04:00
Ryan Houdek 65e8d094ef Merge pull request #3215 from alyssarosenzweig/hack/16k-units
FEXLoader: Query runtime page size
2023-10-23 06:43:43 -07:00
Alyssa Rosenzweig 4c801d594a FEXLoader: Query runtime page size
This lets most of the ASM tests run on 16K Linux hosts which is good because I
have a Mac and I'm bad at computer.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2023-10-23 09:35:22 -04:00
Ryan Houdek d4403edea9 OpcodeDispatcher: Updates COMIS to eliminate scalar moves
This was one of the few things that managed to hit the previously
removed optimization. Just fix the OpcodeDispatcher instead.
2023-10-21 21:33:07 -07:00
Ryan Houdek 06ef012fb2 FEXCore: Fixes bug in vector ZextAndMaskingElimination pass
With the previous RCLSE pass optimization that fixes store->load
forwarding, this pass started optimizing harder.

This hit a bug with this vmov removal that previously didn't get hit.
In particular this would eliminate vmov IR operations even if they were
zero extending a vector.
Since we have dramatically cleaned up the amount of vmov IR operations
we are generating, remove this optimization entirely. In the games I
tested, the only game that hit this "optimization" was Ender Lilies and
it started generating broken code for the single block of instructions
that did.

Adds a unit test for this case just in-case it comes back in the future
for some reason.

Fixes an issue where Ender Lilies would flash the screen to black every
time an enemy hit the player character.
2023-10-21 21:21:14 -07:00
Mai 8f8f37684a Merge pull request #3213 from Sonicadvance1/fix_repres
JITArm64: Fixes bug in rpres scalar operations
2023-10-22 05:05:14 +02:00
Ryan Houdek 7140b8d901 InstCountCI: Update for RPRES fix 2023-10-21 15:29:11 -07:00
Ryan Houdek d5beba9423 JITArm64: Fixes bug in rpres scalar operations
Noticed this during code investigation, these two operations were
swapped.

Would have caused issues if anything supported RPRES today.
2023-10-21 15:24:43 -07:00
Ryan Houdek 826e15aea9 unittests/ASM: Adds dpps/dppd broadcast mask tests
Ensures that the optimization around the broadcast mask is correct.
2023-10-20 18:15:43 +02:00
Ryan Houdek 14e80ce228 InstCountCI: Update for DPPS/DPPD
Adds some new destination broadcast masks to ensure we handle most of
them.
2023-10-20 18:15:43 +02:00
Ryan Houdek 165d3d3d4d Arm64JIT: Fixes VDupElement so it respects 64-bit vector duping
In some cases when we want the upper bits to be zero, this is the
desired behaviour
2023-10-20 18:15:43 +02:00
Ryan Houdek 887200e571 OpcodeDispatcher: Optimize 128-bit DPPS and DPPD
These instructions aren't super amazing due to the fact that they have
both a source mask and a destination duplication mask.

Setup a case where we can generate more optimal code in /most/ cases.

There are a few that still fall down a "bad" path for the result
broadcast but in most cases they are optimal. Still to be seen what
games typically use the broadcast mask as.

AVX in its infinite wisdom expanded DPPS to 256-bit, while leaving DPPD
to only support 128-bit still. This leaves the original implementation
alone for 256-bit DPPS since I don't want to break it.

This is another instruction that gets a free optimization when
SVE-128bit is supported!
2023-10-20 18:02:27 +02:00
Ryan Houdek 2c0bc0654d IR: Adds new VFAddV operation
SVE added this instruction natively, we can take advantage of it on
SVE-128bit systems which is quite nice.

Will be used soon.
2023-10-19 16:38:11 +02:00
Ryan Houdek b3d76bd2f1 IR: Adds DPPS and DPPD source masks
This will get used for these instructions soon
2023-10-19 16:36:19 +02:00
Ryan Houdek 2e694412f4 Merge pull request #3211 from lioncash/ext
VectorOps: Handle SVE VExtr a little better
2023-10-19 16:04:02 +02:00
Lioncache d84577c36c VectorOps: Handle SVE VExtr a little better
If the source registers don't alias the destination, then we can
safely move the lower bits over to it without using a temporary.
2023-10-19 15:11:23 +02:00
Ryan Houdek cf9c2aa72c Merge pull request #3206 from Sonicadvance1/fix_syscall
Linux: Fixes issue with *at syscalls with absolute paths not working
2023-10-19 15:05:34 +02:00
Ryan Houdek 1cb8e4891c Merge pull request #3210 from lioncash/fcadd
VectorOps: Handle SVE VFCADD a little better
2023-10-19 15:05:15 +02:00
Lioncache 24f2796141 VectorOps: Handle SVE VFCADD a little better
If no registers alias, then we can move the first source directly into the
destination and then perform the FCADD operation as opposed to using a
temporary.
2023-10-19 14:48:46 +02:00
Tony Wasserka cb215b5f21 FEXLinuxTests/thunks: Add assume_compatible_data_layout tests 2023-10-19 12:49:00 +02:00
Tony Wasserka 0cf2695772 FEXLinuxTests/thunks: Add tests for opaque types 2023-10-19 12:49:00 +02:00
Tony Wasserka 6a6886305e Thunks/gen: Add assume_compatible/is_opaque annotations
These annotations allow for a given type or parameter to be treated as
"compatible" even if data layout analysis can't infer this automatically.

assume_compatible_data_layout is more powerful than is_opaque, since it
allows for structs containing members of a certain type to be automatically
inferred as "compatible".

Conversely however, is_opaque enforces that the underlying data is never
accessed directly, since non-pointer uses of the type would still be
detected as "incompatible".
2023-10-19 12:49:00 +02:00
Tony Wasserka 5ef7537e61 unittests/thunks: Add ptr_passthrough tests 2023-10-19 12:49:00 +02:00
Tony Wasserka 167fe85cc3 Thunks: Implement ptr_passthrough annotation
This annotation can be used for data types that can't be repacked
automatically even with custom repack annotations. With ptr_passthrough,
the types are wrapped in guest_layout and passed to the host like that.
2023-10-19 12:49:00 +02:00
Tony Wasserka cf65747667 Thunks: Introduce an intermediate guest_layout wrapper to unpack callback arguments
This will be used later to aid automatic struct repacking.
2023-10-19 12:48:59 +02:00
Tony Wasserka 27bb28b47f Thunks: Carry annotations in callback wrappers of host functions
Previously, two functions with the same signature would always be wrapped
in the same logic. This change allows customizing one function with
annotations while leaving the other one unchanged.
2023-10-19 12:48:59 +02:00
Tony Wasserka a00da800e7 Thunks: Rename funcptr_types to thunked_funcptrs
This reflects its purpose slightly better, particularly since future patches
will add more information to this object.
2023-10-19 12:48:59 +02:00
Tony Wasserka bf835e80ac Thunks: Bump compiler requirements to C++20 2023-10-19 12:48:59 +02:00
Tony Wasserka 8f246b206b Merge pull request #3209 from neobrain/refactor_revert_vulkan_reorder 2023-10-19 12:45:14 +02:00
Ryan Houdek 3c5c23bf36 Merge pull request #3208 from lioncash/avg
VectorOps: Handle SVE VURAvg a little better
2023-10-19 12:38:56 +02:00
Tony Wasserka 5bcfaf4b9f Thunks/vulkan: Revert reordering changes from 180d16af7a
These interfere heavily with ongoing work. Let's reapply the reordering
once the dust has settled instead.
2023-10-19 12:31:33 +02:00
Lioncache 1f6c6345d9 VectorOps: Handle SVE VURAvg a little better
We can perform less moves by checking for scenarios where aliasing
occurs. Since addition is commutative (usually, general-case anyway),
order of inputs doesn't strictly matter here.
2023-10-19 12:14:12 +02:00
Ryan Houdek 93792577eb Merge pull request #3207 from lioncash/div
VectorOps: Handle SVE VFDiv a little better
2023-10-19 11:53:45 +02:00
Lioncache 3d23cd5765 VectorOps: Handle SVE VFDiv a little better
In the event no source vectors alias the destination,
we can just move the first source vector into it and
then perform the divide without needing to move afterword.
2023-10-19 11:45:35 +02:00
Ryan Houdek fcc239552c Linux: Fixes issue with *at syscalls with absolute paths not working
When a syscall from the *at series is provided an FD but the path is
absolute then dirfd should be ignored. We weren't correctly doing this.
Now if the path is absolute, but set the argument to the special
AT_FDCWD..
Fixes #3204
2023-10-19 09:48:50 +02:00
Ryan Houdek 8238de024f Merge pull request #3205 from lioncash/max
VectorOps: Handle SVE VSMax/VSMin and VUMax/VUMin paths a little better
2023-10-18 19:24:35 +02:00
Lioncache 39e658f02a VectorOps: Handle more VUMin SVE cases better
We can avoid needing to use movprfx here by moving
directly into the destination when possible and just
doing the UMIN directly
2023-10-18 18:48:13 +02:00
Lioncache e89dd27f2a VectorOps: Handle more VSMin SVE cases better
We can avoid needing to use movprfx here by moving
directly into the destination when possible and just
doing the SMIN directly.
2023-10-18 18:48:13 +02:00
Lioncache f85fae0041 VectorOps: Handle more VUMax SVE cases better
We can avoid needing to use movprfx here by moving
directly into the destination when possible and just
doing the UMAX directly.

Also expands the unsigned max tests to test values with
the sign bit set to ensure all behavior is caught.
2023-10-18 18:48:12 +02:00
Lioncache 65eec673fc VectorOps: Handle more VSMax SVE cases better
Since SMAX performs a comparison and returns the max value regardless
of how the operands are provided, we can check for when the second
input aliases the destination.
2023-10-18 18:48:03 +02:00
Ryan Houdek 5c93a085d2 Merge pull request #3203 from lioncash/movs
OpcodeDispatcher: Handle SSE vector moves into themselves a little better
2023-10-18 16:28:45 +02:00
Lioncache 4b356a7c2c OpcodeDispatcher: Have MOVNTSD go down the non-temporal path
For some reason this was using the regular unaligned path.
2023-10-18 14:59:02 +02:00
Lioncache 2b67f87054 OpcodeDispatcher: Handle SSE vector moves into themselves a little better
Obviously, it's silly to do this, but we should still be generating
optimal code for this case (which is none at all).
2023-10-18 14:58:57 +02:00
Ryan Houdek 1ea40ae676 Merge pull request #3201 from neobrain/fix_flt_thunks_64bit_only
FEXLinuxTests: Temporarily limit thunk test execution to 64-bit guests
2023-10-18 12:40:29 +02:00
Ryan Houdek e0ef32e0bf Merge pull request #3202 from Sonicadvance1/oopsies_vulkan
Thunks: Oops deleted an entry point
2023-10-18 12:38:05 +02:00
Ryan Houdek a2b53c8eb0 Thunks: Oops deleted an entry point
Moving some entries around I managed to delete one.
Fixes Vulkan thunks.
2023-10-18 12:21:28 +02:00
Tony Wasserka 21b6cccb4e FEXLinuxTests: Temporarily limit thunk test execution to 64-bit guests
Thunking isn't fully functional on 32-bit guests currently, so non-trivial
tests would currently hang in that context.
2023-10-18 12:09:10 +02:00
Tony Wasserka d539829251 FEXLinuxTests: Drop .32/.64 suffixes from test names 2023-10-18 12:09:10 +02:00
Ryan Houdek ef321e4bf8 Merge pull request #3200 from lioncash/mov
OpcodeDispatcher: Remove unnecessary 128-bit truncating moves from StoreResult
2023-10-17 12:12:48 +02:00
Lioncache 47a0f14537 OpcodeDispatcher: Remove unnecessary 128-bit truncating moves from StoreResult
Removes the truncating move that we perform inside the StoreResult
function and instead delegates the responsibility to the instruction
implementations themselves.

This removes a lot of redundant moves that occur on 128-bit variants
of AVX instructions.

Also fixes a weird case where we were handling 128-bit SVE
in VBroadcastFromMem when we already have AdvSIMD instructions
that will perfom the zero-extension behavior for us.
2023-10-17 11:07:04 +02:00
Ryan Houdek 6d39f369b0 Merge pull request #3199 from lioncash/loadops
OpcodeDispatcher: Put extra LoadSource options in a struct
2023-10-16 09:42:27 +02:00
Lioncache 2304cfc530 OpcodeDispatcher: Remove prefixing from MemoryAccessType enum
Since this is an enum class, we don't need to add a prefix.
2023-10-16 03:10:33 +02:00
Lioncache 1a39de4509 OpcodeDispatcher: Put extra LoadSource options in a struct
Allows for easier expansion without needing to expand the function definitons.

Also makes a few usages significantly less verbose and makes specifying
options a little more declarative, rather than having to memorize what
each argument is specifying.
2023-10-15 21:18:00 +02:00
Ryan Houdek efb479f88f Merge pull request #3198 from lioncash/rorx
OpcodeDispatcher: Remove redundant moves from rorx
2023-10-15 17:13:00 +02:00
Lioncache b27bf43901 OpcodeDispatcher: Remove redundant moves from rorx
By allowing junk in the upper bits, we can avoid an unnecessary move,
since we'll be ignoring them in the following ROR instruction anyway.
2023-10-15 17:05:15 +02:00
Ryan Houdek c612fa8f2f Merge pull request #3196 from lioncash/fmt 2023-10-13 10:44:43 -07:00
Lioncache 4ccc40f697 Externals: Update fmt from 10.1.0 to 10.1.1
Notably this bugfix version also introduces support for formatting
std::atomic types and std::atomic_flag.

Also, of course keeps our tracked external up to date.
2023-10-13 13:31:49 -04:00
Ryan Houdek cb53a704ba Merge pull request #3195 from Sonicadvance1/fix_missing_vulkan
Thunks: Fixes missing vulkan definitions
2023-10-11 18:51:52 -07:00
Alyssa Rosenzweig 483423674a Merge pull request #3194 from Sonicadvance1/instcountci_multiinst_tests
InstCountCI: Adds some multi instruction tests
2023-10-11 05:06:55 -04:00
Ryan Houdek 180d16af7a Thunks: Fixes missing vulkan definitions
A couple of games were hitting these. Not sure how they were missed in
PR #3159 but adds the missing one.

Small rearrangement to make this easier as well. Hopefully thunk stuff
lands sooner rather than later to automate this for Vulkan.

Maybe `-isystem` instead of `-I` needs to be used unlike what #2076,
might depend on what is installed on the host system.
2023-10-11 01:49:42 -07:00
Ryan Houdek 1acc038826 InstCountCI: Adds some multi instruction tests
Some simple tests to showcase instructions that we can optimize.
- Back to back pushes could be optimized
- Back to back scalar vector operations can be optimized
- Show with AFP that back to back scalar is already optimal
   - Also ensures we don't break this stage.
2023-10-10 16:45:39 -07:00
Alyssa Rosenzweig cc558fd5dc Merge pull request #3192 from Sonicadvance1/optimize_small_push
OpcodeDispatcher: Optimizes < 32-bit register push
2023-10-10 17:10:34 -04:00
Ryan Houdek 8f04223193 InstCountCI: Adds push changed results
Removes an accidental Test.json file that I use locally.
2023-10-10 12:06:36 -07:00
Ryan Houdek 4be649c44e OpcodeDispatcher: Optimizes < 32-bit register push
We don't need zero in the upper bits for a push.
Makes a couple variants optimal.

Adds missing tests to the 32-bit file, since only 32-bit can push a
32-bit register.
2023-10-10 12:04:41 -07:00
Ryan Houdek 6253f4f708 Merge pull request #3186 from Sonicadvance1/afp_support
IR: Adds scalar vector insert operations
2023-10-10 11:28:22 -07:00
Ryan Houdek cc2eef619c InstCountCI: Update for AFP optimizations
A bunch of random instructions have converted to be optimal in a vacuum.
2023-10-10 03:44:59 -07:00
Ryan Houdek 3bff42e6a7 OpcodeDispatcher: Wire up support for the new scalar insert operations 2023-10-10 03:44:58 -07:00
Ryan Houdek 2671246fef IR: Adds scalar vector insert operations
These IR operations are required to support AFP's NEP mode which does
vector insert in to the destination register. Additionally it gives us
tracking information to allow optimizing out redundant inserts on
devices that don't support AFP natively.

In order to match x86 semantics we need to support binary and unary
scalar operations that do a final insert in to a vector. With optional
zeroing of the top 128-bits for AVX variants.

A tricky thing is that in binary operations this means that the
destination and first source have an intrinsically linked property
depending on if it is SSE or AVX.

SSE example:
- addss xmm0, xmm1
   - xmm0 is both the destination and the first source.
   - This means xmm0[31:0] = xmm0[31:0] + xmm1[31:0]
   - Bits [127:32] are UNMODIFIED.

FEX's JIT jumps through some hoops so that if the destination register
equals the first source register, then it hits the optimal path the
AFP.NEP will insert in to the result. AVX throws a small wrench in to
this due to changed behaviour

AVX example:
- vaddss xmm0, xmm1, xmm2
  - xmm0 is ONLY the destination, xmm1 and xmm2 are the sources
  - This operation copies the bits above the scalar result from the
    first source (xmm1).
  - Additionally this will zero bits above the original 128-bit xmm
    register.
  - xmm0[31:0] = xmm1[31:0] + xmm2[31:0]
  - xmm0[127:32] = xmm1[127:32]
  - ymm0[255:127] = 0

This causes these instructions to support a fairly large table depending
on if the instruction is an SSE or AVX instruction, plus if the host CPU
supports AFP or not.

So while fairly complex, it's handling all the edge cases and gives us
optimization opportunities as we move forward. Currently on non-AFP
supporting devices this has a minor benefit that these IR operations
remove one temporary register, lowering the Register Allocation
overhead.

In the coming weeks I am likely to introduce an optimization pass that
removes redundant inserts because FEX currently does /really/ badly with
scalar code loops.

Needs #3184 merged first.
2023-10-10 03:17:19 -07:00
Ryan Houdek 8cb8f090dd Arm64Emitter: enable/disable AFP on Fill/Spill
When FEX is in the JIT we need to make sure to enable NEP and AH and
then disable when leaving.

Explicitly disabled when the vixl simulator is used since even
attempting to set the bits will cause it to fault out. Ensures
InstCountCI keeps working.
2023-10-10 03:17:18 -07:00
Ryan Houdek a37d89a7d5 HostFeatures: Disable AFP until verified that it is working
Need to audit scalar instruction usage to ensure all uses are okay with
garbage in the upper bits.
2023-10-10 03:17:18 -07:00
Ryan Houdek 252d7712ea Arm64: Save if the host supports AFP 2023-10-10 03:17:18 -07:00
Ryan Houdek c548625fbe InstCountCI: Update tests for disabling AFP
Doesn't change behaviour yet, just prep work.
2023-10-10 03:17:18 -07:00
Ryan Houdek f036a0b84f Merge pull request #3191 from Sonicadvance1/instcountci_multiple
InstCountCI: Support multiple instructions in the tests
2023-10-10 02:53:14 -07:00
Ryan Houdek 2e1389b25e Merge pull request #3184 from Sonicadvance1/armemitter_sized_scalars
ArmEmitter: Adds sized Scalar 1 source and 2 source helpers
2023-10-10 02:53:06 -07:00
Alyssa Rosenzweig a5f82a57fa Merge pull request #3190 from Sonicadvance1/atomic_instcountci
InstCountCI: Adds missing atomic tests
2023-10-10 05:22:36 -04:00
Ryan Houdek cd83d3eb24 InstCountCI: Support multiple instructions in the tests
There are some cases where we want to test multiple instructions where
we can do optimizations that would overwise be hard to see.

eg:
```asm
; Can be optimized to a single stp
push eax
push ebx

; Can remove half of the copy since we know the direction
cld
rep movsb

; Can remove a redundant insert
addss xmm0, xmm1
addss xmm0, xmm2
```

This lets us have arbitrary sized code in instruction count CI, with the
original json key becoming only a label if the instruction array is
provided.

There are still some major limitations to this, instructions that
generate side-effects might have "garbage" after the end of the block
that isn't correctly accounted for. So care must be taken.

Example in the json
```json
"push ax, bx": {
  "ExpectedInstructionCount": 4,
  "Optimal": "No",
  "Comment": "0x50",
  "x86Insts": [
    "push ax",
    "push bx"
  ],
  "ExpectedArm64ASM": [
    "uxth w20, w4",
    "strh w20, [x8, #-2]!",
    "uxth w20, w7",
    "strh w20, [x8, #-2]!"
  ]
}
```
2023-10-09 21:49:53 -07:00
Ryan Houdek 93ab8ab23c InstCountCI: Adds missing atomic tests
This adds all the missing atomic tests in to their own tests files.
This includes all of them except a few choice ones that are in their
original files.

- BTC, BTR, BTS  are in their Secondary/SecondaryGroup files
- CMPXCHG, CMPXCHG8B, CMPXCHG16B are in their Secondary/SecondaryGroup
  files
   - These always imply lock semantics even without the prefix.
2023-10-09 21:18:08 -07:00
Ryan Houdek 462fff2c67 Merge pull request #3189 from Sonicadvance1/remove_warnings_15
FEXCore: Removes a warning about assume discarding side-effects
2023-10-09 17:26:14 -07:00
Alyssa Rosenzweig 8dab35cbf8 Merge pull request #3188 from Sonicadvance1/reconstruct_flags_naming
FEXCore: Renames raw FLAGS location names to signify they can't be used directly
2023-10-09 19:41:09 -04:00
Ryan Houdek a1a479e69f FEXCore: Removes a warning about assume discarding side-effects 2023-10-09 16:04:57 -07:00
Ryan Houdek 6403290019 FEXCore: Renames raw FLAGS location names to signify they can't be used directly
Six of the EFLAGS can't be used directly in a bitmask because they are
either contained in a different flags location or has multiple bits
stored in it.

SF, ZF, CF, OF are stored in ARM's NZCV format in offset 24.
PF calculation is deferred but stored in the regular offset.
AF is also deferred in relation to the PF but stored in the regular
offset.

These /need/ to be reconstructed using the `ReconstructCompactedEFLAGS`
function when wanting to read the EFLAGS.

When setting these flags they /need/ to be set using
`SetFlagsFromCompactedEFLAGS`.

If either of these functions are not used when managing EFLAGs then the
internal representation will get mangled and the state will be
corrupted.

Having a little `_RAW` on these to signify that these aren't just
regular single bit representations like the other flags in EFLAGS should
make us puzzle about this issue before writing more broken code that
tries accessing it directly.
2023-10-08 11:51:11 -07:00
Ryan Houdek 580bd50a00 unittests/ASM: Removes eflags comparison option
This was not used and is also broken.
2023-10-08 11:51:11 -07:00
Ryan Houdek b2a8b0ca12 Merge pull request #3187 from Sonicadvance1/implement_rpres
FEXCore: Implements support for RPRES
2023-10-08 09:57:27 -07:00
Ryan Houdek f78bdf0852 unittests/Emitter: Adds sized scalar unittests. 2023-10-08 09:48:37 -07:00
Ryan Houdek 5652eb4c5d ARMEmitter: Removes templated ptrue/ptrues
Non-templated version exists and templated version gets us nothing.
2023-10-07 23:51:32 -07:00
Ryan Houdek a52bb47551 unittests: Update for rpres optimization 2023-10-07 23:22:51 -07:00
Ryan Houdek 22590dde77 FEXCore: Implements support for RPRES
This allows us to use reciprocal instructions which matches precision of
what x86 expects rather than converting everything to float divides.

Currently no hardware supports this, and even the upcoming X4/A720/A520
won't support it, but it was trivial to implement so wire it up.
2023-10-07 23:13:47 -07:00
Ryan Houdek 6543a80ff9 Merge pull request #3185 from Sonicadvance1/ir_dispatcher_emit
FEXCore/IR: Changes over to automated IR dispatch generation
2023-10-07 21:21:44 -07:00
Ryan Houdek 9c36d1061b Merge pull request #3182 from Sonicadvance1/instcountci_stacking_test_names
InstCountCI: Fixes recursive tests with same filename
2023-10-07 21:21:10 -07:00
Alyssa Rosenzweig 5a3cc7b469 Merge pull request #3183 from Sonicadvance1/instcountci_support_afp_override
InstCountCI: Support overriding AFP features
2023-10-07 19:57:33 -04:00
Ryan Houdek 4cff3e5f1f FEXCore/IR: Changes over to automated IR dispatch generation
Suggested by Alyssa. Adding an IR operation can be a little tedious
since you need to add the definition to JIT.cpp for the dispatch switch,
JITClass.h for the function declared, and then actually defining the
implementation in the correct file.

Instead support the common case where an IR operation just gets
dispatched through to the regular handler. This lets the developer just
put the function definition in to the json and the relevent cpp file and
it just gets picked up.

Some minor things:
- Needs to support dynamic dispatch for {Load,Store}Register and
  {Load,Store}Mem
   - This is just a bool in the json
- It needs to not output JIT dispatch for some IR operations
   - SSE4.2 string instructions and x87 operations
   - These go down the "Unhandled" path
- Needs to support a Dispatcher function override
   - This is just for handling NoOp IR operations that get used for
     other reasons.
- Finally removes VSMul and VUMul, consolidating to VMul
   - Unlike V{U,S}Mull, signed or unsigned doesn't change behaviour here
- Fixed a couple random handler names not matching the IR operation
  name.
2023-10-07 15:01:47 -07:00
Ryan Houdek a1eb571630 ArmEmitter: Adds sized Scalar 1 source and 2 source helpers
Removes the need for an annoying switch statement with scalar operations
for the most part.
2023-10-07 11:51:32 -07:00
Ryan Houdek 559cf6491a InstCountCI: Support overriding AFP features
Also disable AFP under the vixl simulator by default since it doesn't support it.
2023-10-07 11:48:42 -07:00
Ryan Houdek 4bdda1eeb5 InstCountCI: Fixes recursive tests with same filename
This will be used to move AFP tests to a sub-directory
2023-10-07 11:47:16 -07:00
Mai fc70fc3506 Merge pull request #3179 from Sonicadvance1/support_hostfeature_crypto
FEXCore: Support crypto extensions in HostFeatures override
2023-10-06 16:01:59 -04:00
Mai 26ee63cc24 Merge pull request #3181 from Sonicadvance1/remove_spurious_license
External: Remove a spurious license
2023-10-06 15:59:41 -04:00
Ryan Houdek 0092ea7c0b External: Remove a spurious license
This doesn't exist anymore
2023-10-06 09:37:17 -07:00
Ryan Houdek 439a3b9c3a HostFeatures: Use a define 2023-10-06 09:33:41 -07:00
Alyssa Rosenzweig b4ddf36582 Merge pull request #3180 from Sonicadvance1/remove_warnings_14
Linux: Fixes warning in 32-bit clock_settime
2023-10-06 08:13:52 -04:00
Ryan Houdek 12c44f26e5 Linux: Fixes warning in 32-bit clock_settime
This syscall requires a valid pointer otherwise it returns EFAULT.
When going through the glibc helper it can crash before reaching the raw
syscall even.
2023-10-05 17:44:33 -07:00
Ryan Houdek 5b7ba06d5c FEXCore: Support crypto extensions in HostFeatures override
Enables in InstCountCI so Pi users can run InstCountCI can run the tests
without breaking on crypto operations.

When crypto is enabled or disabled just wholesale change AES, CRC32, and
PMULL 128-bit in one step. We don't really care about partial support
here.
2023-10-05 17:41:08 -07:00
1005 changed files with 426290 additions and 128134 deletions

No files matched your search

+109
View File
@@ -0,0 +1,109 @@
Language: Cpp
BasedOnStyle: WebKit
AccessModifierOffset: -2
AlignAfterOpenBracket: Align
AlignArrayOfStructures: None
AlignConsecutiveAssignments: None
AlignConsecutiveBitFields: Consecutive
AlignConsecutiveDeclarations: None
AlignConsecutiveMacros: None
AlignEscapedNewlines: Left
AlignOperands: Align
AlignTrailingComments: true
AllowAllParametersOfDeclarationOnNextLine: false
AllowShortCaseLabelsOnASingleLine: true
AllowShortEnumsOnASingleLine: true
AllowShortFunctionsOnASingleLine: Empty
AllowShortIfStatementsOnASingleLine: WithoutElse
AllowShortLambdasOnASingleLine: Inline
AlwaysBreakAfterDefinitionReturnType: None
AlwaysBreakAfterReturnType: None
AlwaysBreakBeforeMultilineStrings: false
AlwaysBreakTemplateDeclarations: true
AttributeMacros:
- JEMALLOC_NOTHROW
- FEX_ALIGNED
- FEX_ANNOTATE
- FEX_DEFAULT_VISIBILITY
- FEX_NAKED
- FEX_PACKED
- FEXCORE_PRESERVE_ALL_ATTR
- GLIBC_ALIAS_FUNCTION
BinPackArguments: true
BinPackParameters: true
BitFieldColonSpacing: Both
BreakAfterAttributes: Always # clang 16 required
BreakBeforeBraces: Attach
BreakBeforeBinaryOperators: None
BreakBeforeInlineASMColon: OnlyMultiline # clang 16 required
BreakBeforeTernaryOperators: false
BreakConstructorInitializers: BeforeComma
BreakInheritanceList: BeforeColon
ColumnLimit: 140
CompactNamespaces: false
ConstructorInitializerIndentWidth: 2
ContinuationIndentWidth: 2
Cpp11BracedListStyle: true
DerivePointerAlignment: false
EmptyLineAfterAccessModifier: Leave
EmptyLineBeforeAccessModifier: Leave
ExperimentalAutoDetectBinPacking: false
FixNamespaceComments: true
IncludeBlocks: Preserve
IndentAccessModifiers: false
IndentCaseBlocks: false
IndentCaseLabels: false
IndentExternBlock: AfterExternBlock
IndentGotoLabels: false
IndentPPDirectives: None
IndentRequires: false
IndentWidth: 2
InsertBraces: true
KeepEmptyLinesAtTheStartOfBlocks: true
LambdaBodyIndentation: OuterScope
LineEnding: LF # clang 16 required
MaxEmptyLinesToKeep: 2
NamespaceIndentation: Inner
QualifierAlignment: Left
PackConstructorInitializers: Never
PenaltyBreakAssignment: 2
PenaltyBreakBeforeFirstCallParameter: 2
PenaltyBreakOpenParenthesis: 2
PenaltyBreakString: 10
PenaltyBreakTemplateDeclaration: 8
PenaltyExcessCharacter: 2
PenaltyReturnTypeOnItsOwnLine: 16
PointerAlignment: Left
RemoveBracesLLVM: false
ReferenceAlignment: Left
ReflowComments: true
RequiresClausePosition: WithPreceding
SeparateDefinitionBlocks: Leave
SortIncludes: Never
SpaceAfterCStyleCast: false
SpaceAfterLogicalNot: false
SpaceAfterTemplateKeyword: false
SpaceAroundPointerQualifiers: Default
SpaceBeforeAssignmentOperators: true
SpaceBeforeCaseColon: false
SpaceBeforeCpp11BracedList: true
SpaceBeforeInheritanceColon: true
SpaceBeforeParens: Custom
SpaceBeforeParensOptions:
AfterControlStatements: true
AfterFunctionDeclarationName: false
AfterFunctionDefinitionName: false
AfterOverloadedOperator: false
AfterRequiresInClause: true
BeforeNonEmptyParentheses: false
SpaceBeforeRangeBasedForLoopColon: true
SpaceBeforeSquareBrackets: false
SpaceInEmptyBlock: false
SpaceInEmptyParentheses: false
SpacesBeforeTrailingComments: 1
SpacesInAngles: Leave
SpacesInCStyleCastParentheses: false
SpacesInConditionalStatement: false
SpacesInParentheses: false
Standard: c++20
UseTab: Never
+12
View File
@@ -0,0 +1,12 @@
# This file is used to ignore files and directories from clang-format
# Ignore all files in the External directory
External/*
# SoftFloat-3e code doesn't belong to us
FEXCore/Source/Common/SoftFloat-3e/*
Source/Common/cpp-optparse/*
# Files with human-indented tables for readability - don't mess with these
FEXCore/Source/Interface/Core/X86Tables/*
+15
View File
@@ -0,0 +1,15 @@
# Since version 2.23 (released in August 2019), git-blame has a feature
# to ignore or bypass certain commits.
#
# This file contains a list of commits that are not likely what you
# are looking for in a blame, such as mass reformatting or renaming.
# You can set this file as a default ignore file for blame by running
# the following command.
#
# $ git config blame.ignoreRevsFile .git-blame-ignore-revs
# Whole tree reformat PR#3571
2b4ec88daebd35fefb5bf5c73d7fc2b4155771ed
# Second reformat to find fixed point PR#3577
905aa935f5ce344a48ef4d5edab3c31efa8d793e
@@ -37,7 +37,6 @@ If applicable, add screenshots and video to help explain your problem.
**Additional context**
- Is this an x86 or x86-64 game: [x86/x86-64/Both]
- Does this reproduce on x86-64 host with FEX: [Yes/No/Untested]
- Does this reproduce on AArch64 with Radeon/Intel/Nvidia: [Yes/No/Untested]
- Is this a Vulkan game: [Yes/No/Unknown]
- If Yes, What is your Vulkan driver:
+1 -15
View File
@@ -13,8 +13,6 @@ env:
BUILD_TYPE: Release
CC: clang
CXX: clang++
FEX_FORCE32BITALLOCATOR: 1
FEX_ENABLEAVX: 1
jobs:
build_plus_test:
@@ -78,18 +76,6 @@ jobs:
shell: bash
run: cmake --build . --config $BUILD_TYPE --target install
- name: IR Tests
working-directory: ${{runner.workspace}}/build
shell: bash
# Execute the unit tests
run: cmake --build . --config $BUILD_TYPE --target ir_tests
- name: IR Test Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_IR.log || true
- name: gcc target tests 64
working-directory: ${{runner.workspace}}/build
shell: bash
@@ -250,7 +236,7 @@ jobs:
working-directory: ${{runner.workspace}}/build
# Cap out the log files at 20M in case something crash spins and dumps fault text
# ASM tests get quite close to 10MB
run: truncate --size=<20M ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
run: truncate --size="<20M" ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
- name: Remove old SHM regions
if: ${{ always() }}
+1 -15
View File
@@ -20,8 +20,6 @@ env:
BUILD_TYPE: Release
CC: clang
CXX: clang++
FEX_FORCE32BITALLOCATOR: 1
FEX_ENABLEAVX: 1
jobs:
glibc_fault_test:
@@ -85,18 +83,6 @@ jobs:
shell: bash
run: cmake --build . --config $BUILD_TYPE --target install
- name: IR Tests
working-directory: ${{runner.workspace}}/build
shell: bash
# Execute the unit tests
run: cmake --build . --config $BUILD_TYPE --target ir_tests
- name: IR Test Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_IR.log || true
- name: gcc target tests 64
working-directory: ${{runner.workspace}}/build
shell: bash
@@ -184,7 +170,7 @@ jobs:
working-directory: ${{runner.workspace}}/build
# Cap out the log files at 20M in case something crash spins and dumps fault text
# ASM tests get quite close to 10MB
run: truncate --size=<20M ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
run: truncate --size="<20M" ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
- name: Remove old SHM regions
if: ${{ always() }}
+1 -2
View File
@@ -13,7 +13,6 @@ env:
BUILD_TYPE: Release
CC: clang
CXX: clang++
FEX_ENABLEAVX: 1
jobs:
hostrunner_tests:
@@ -90,7 +89,7 @@ jobs:
working-directory: ${{runner.workspace}}/build
# Cap out the log files at 20M in case something crash spins and dumps fault text
# ASM tests get quite close to 10MB
run: truncate --size=<20M ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
run: truncate --size="<20M" ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
- name: Set runner name
if: ${{ always() }}
+12 -3
View File
@@ -13,7 +13,6 @@ env:
BUILD_TYPE: Release
CC: clang
CXX: clang++
FEX_ENABLEAVX: 1
jobs:
instcountci_tests:
@@ -56,6 +55,16 @@ jobs:
# We'll use this as our working directory for all subsequent commands
run: cmake -E make_directory ${{runner.workspace}}/build
- name: Set vixl_sim x86
if: matrix.arch[1] == 'x64'
run: |
echo "VIXL_SIM_ENABLED=True" >> $GITHUB_ENV
- name: Set vixl_sim Arm64
if: matrix.arch[1] == 'ARM64'
run: |
echo "VIXL_SIM_ENABLED=False" >> $GITHUB_ENV
- name: Configure CMake
# Use a bash shell so we can use the same syntax for environment variable
# access regardless of the host operating system
@@ -64,7 +73,7 @@ jobs:
# Note the current convention is to use the -S and -B options here to specify source
# and build directories, but this is only available with CMake 3.13 and higher.
# The CMake binaries on the Github Actions machines are (as of this writing) 3.12
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=$BUILD_TYPE -G Ninja -DENABLE_VIXL_SIMULATOR=False -DENABLE_VIXL_DISASSEMBLER=True -DENABLE_LTO=False -DENABLE_ASSERTIONS=True -DENABLE_X86_HOST_DEBUG=True
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=$BUILD_TYPE -G Ninja -DENABLE_VIXL_SIMULATOR=$VIXL_SIM_ENABLED -DENABLE_VIXL_DISASSEMBLER=True -DENABLE_LTO=False -DENABLE_ASSERTIONS=True -DENABLE_X86_HOST_DEBUG=True
- name: Build
working-directory: ${{runner.workspace}}/build
@@ -111,7 +120,7 @@ jobs:
working-directory: ${{runner.workspace}}/build
# Cap out the log files at 20M in case something crash spins and dumps fault text
# ASM tests get quite close to 10MB
run: truncate --size=<20M ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
run: truncate --size="<20M" ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
- name: Set runner name
if: ${{ always() }}
-1
View File
@@ -10,7 +10,6 @@ on:
env:
BUILD_TYPE: Debug
FEX_ENABLEAVX: 1
jobs:
mingw_build:
+76
View File
@@ -0,0 +1,76 @@
# Inspired by LLVM's pr-code-format.yml at
# https://github.com/llvm/llvm-project/blob/main/.github/workflows/pr-code-format.yml
name: "Check code formatting"
on:
pull_request:
branches:
- main
jobs:
code_formatter:
runs-on: [self-hosted, X64]
if: github.repository == 'FEX-Emu/FEX'
steps:
- name: Fetch FEX sources
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout through merge base
uses: rmacklin/fetch-through-merge-base@v0
timeout-minutes: 3
with:
base_ref: ${{ github.event.pull_request.base.ref }}
head_ref: ${{ github.event.pull_request.head.sha }}
deepen_length: 500
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v39
with:
separator: ","
skip_initial_fetch: true
- name: "Listed files"
env:
CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
run: |
echo "Formatting files:"
echo "$CHANGED_FILES"
- name: Check for correct clang-format version
run: clang-format --version | grep -qF '16.0.6'
- name: Check git-clang-format-16 exists
run: which git-clang-format-16
- name: Setup Python env
uses: actions/setup-python@v4
with:
python-version: '3.11'
cache: 'pip'
cache-dependency-path: './External/code-format-helper/requirements_formatting.txt'
- name: Install python dependencies
run: pip install -r ./External/code-format-helper/requirements_formatting.txt
- name: Run code formatter
env:
CLANG_FORMAT_PATH: 'git-clang-format-16'
GITHUB_PR_NUMBER: ${{ github.event.pull_request.number }}
START_REV: ${{ github.event.pull_request.base.sha }}
END_REV: ${{ github.event.pull_request.head.sha }}
CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
# TODO(pmatos): Once we adopt v18, we should be able
# to take advantage of the new --diff_from_common_commit option
# explicitly in code-format-helper.py and not have to diff starting at
# the merge base.
run: |
python ./External/code-format-helper/code-format-helper.py \
--repo "FEX-emu/FEX" \
--issue-number $GITHUB_PR_NUMBER \
--start-rev $(git merge-base $START_REV $END_REV) \
--end-rev $END_REV \
--changed-files "$CHANGED_FILES"
+1 -14
View File
@@ -13,7 +13,6 @@ env:
BUILD_TYPE: Release
CC: clang
CXX: clang++
FEX_ENABLEAVX: 1
jobs:
vixl_simulator:
@@ -100,25 +99,13 @@ jobs:
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_ASM128bit.log || true
- name: IR Tests
working-directory: ${{runner.workspace}}/build
shell: bash
# Execute the unit tests
run: cmake --build . --config $BUILD_TYPE --target ir_tests
- name: IR Test Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_IR.log || true
- name: Truncate test results
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
# Cap out the log files at 20M in case something crash spins and dumps fault text
# ASM tests get quite close to 10MB
run: truncate --size=<20M ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
run: truncate --size="<20M" ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
- name: Set runner name
if: ${{ always() }}
+6 -6
View File
@@ -15,19 +15,19 @@
path = External/tiny-json
url = https://github.com/Sonicadvance1/tiny-json.git
[submodule "External/xbyak"]
shallow = true
shallow = true
path = External/xbyak
url = https://github.com/FEX-Emu/xbyak.git
url = https://github.com/herumi/xbyak.git
[submodule "External/fex-posixtest-bins"]
shallow = true
shallow = true
path = External/fex-posixtest-bins
url = https://github.com/FEX-Emu/fex-posixtest-bins.git
[submodule "External/fex-gvisor-tests-bins"]
shallow = true
shallow = true
path = External/fex-gvisor-tests-bins
url = https://github.com/FEX-Emu/fex-gvisor-tests-bins.git
[submodule "External/fex-gcc-target-tests-bins"]
shallow = true
shallow = true
path = External/fex-gcc-target-tests-bins
url = https://github.com/FEX-Emu/fex-gcc-target-tests-bins.git
[submodule "External/jemalloc"]
@@ -41,7 +41,7 @@
url = https://github.com/FEX-Emu/drm-headers.git
[submodule "External/xxhash"]
path = External/xxhash
url = https://github.com/FEX-Emu/xxHash.git
url = https://github.com/Cyan4973/xxHash.git
[submodule "External/Catch2"]
path = External/Catch2
url = https://github.com/catchorg/Catch2.git
+42 -74
View File
@@ -9,7 +9,6 @@ option(BUILD_FEX_LINUX_TESTS "Build FEXLinuxTests, requires x86 compiler" FALSE)
option(BUILD_THUNKS "Build thunks" FALSE)
option(BUILD_FEXCONFIG "Build FEXConfig, requires SDL2 and X11" TRUE)
option(ENABLE_CLANG_THUNKS "Build thunks with clang" FALSE)
option(ENABLE_CLANG_FORMAT "Run clang format over the source" FALSE)
option(ENABLE_IWYU "Enables include what you use program" FALSE)
option(ENABLE_LTO "Enable LTO with compilation" TRUE)
option(ENABLE_XRAY "Enable building with LLVM X-Ray" FALSE)
@@ -26,7 +25,6 @@ option(ENABLE_OFFLINE_TELEMETRY "Enables FEX offline telemetry" TRUE)
option(ENABLE_COMPILE_TIME_TRACE "Enables time trace compile option" FALSE)
option(ENABLE_LIBCXX "Enables LLVM libc++" FALSE)
option(ENABLE_CCACHE "Enables ccache for compile caching" TRUE)
option(ENABLE_TERMUX_BUILD "Forces building for Termux on a non-Termux build machine" FALSE)
option(ENABLE_VIXL_SIMULATOR "Forces the FEX JIT to use the VIXL simulator" FALSE)
option(ENABLE_VIXL_DISASSEMBLER "Enables debug disassembler output with VIXL" FALSE)
option(COMPILE_VIXL_DISASSEMBLER "Compiles the vixl disassembler in to vixl" FALSE)
@@ -45,6 +43,14 @@ if (NOT CONTAINS_MINGW EQUAL -1)
set (ENABLE_JEMALLOC FALSE)
endif()
if (NOT MINGW_BUILD)
message (STATUS "Clang version ${CMAKE_CXX_COMPILER_VERSION}")
set (CLANG_MINIMUM_VERSION 12.0)
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS ${CLANG_MINIMUM_VERSION})
message (FATAL_ERROR "Clang version too old for FEX. Need at least ${CLANG_MINIMUM_VERSION} but has ${CMAKE_CXX_COMPILER_VERSION}")
endif()
endif()
if (ENABLE_FEXCORE_PROFILER)
add_definitions(-DENABLE_FEXCORE_PROFILER=1)
string(TOUPPER "${FEXCORE_PROFILER_BACKEND}" FEXCORE_PROFILER_BACKEND)
@@ -112,6 +118,12 @@ else()
endif()
if (CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
option(ENABLE_X86_HOST_DEBUG "Enables compiling on x86_64 host" FALSE)
if (NOT ENABLE_X86_HOST_DEBUG)
message(FATAL_ERROR
" FEX-Emu doesn't support compiling for x86-64 hosts!"
" This is /only/ a supported configuration for FEX CI and nothing else!")
endif()
set(_M_X86_64 1)
add_definitions(-D_M_X86_64=1)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcx16")
@@ -122,6 +134,14 @@ if (CMAKE_SYSTEM_PROCESSOR MATCHES "^aarch64|^arm64|^armv8\.*")
add_definitions(-D_M_ARM_64=1)
endif()
if (CMAKE_SYSTEM_PROCESSOR MATCHES "^arm64ec")
set(_M_ARM_64EC 1)
add_definitions(-D_M_ARM_64EC=1)
# Required as FEX is not allowed to lock the CRT heap lock during compilation or callbacks
set(ENABLE_JEMALLOC TRUE)
endif()
if (ENABLE_CCACHE)
find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
@@ -158,18 +178,6 @@ if (NOT ENABLE_OFFLINE_TELEMETRY)
add_definitions(-DFEX_DISABLE_TELEMETRY=1)
endif()
if(DEFINED ENV{TERMUX_VERSION} OR ENABLE_TERMUX_BUILD)
add_definitions(-DTERMUX_BUILD=1)
set(TERMUX_BUILD 1)
# Termux doesn't support Jemalloc due to bad interactions between emutls, jemalloc, and scudo
set(ENABLE_JEMALLOC FALSE)
# Termux builds can't rely on X11 packages
# SDL2 isn't even compiled with GL support so our GUIs wouldn't even work
set(BUILD_FEXCONFIG FALSE)
endif()
if (ENABLE_ASAN)
add_definitions(-DENABLE_ASAN=1)
add_compile_options(-fno-omit-frame-pointer -fsanitize=address -fsanitize-address-use-after-scope)
@@ -232,15 +240,15 @@ endif()
find_package(PkgConfig REQUIRED)
find_package(Python 3.0 REQUIRED COMPONENTS Interpreter)
add_subdirectory(External/xxhash/)
include_directories(External/xxhash/)
set(XXHASH_BUNDLED_MODE TRUE)
set(XXHASH_BUILD_XXHSUM FALSE)
set(BUILD_SHARED_LIBS OFF)
add_subdirectory(External/xxhash/cmake_unofficial/)
add_definitions(-Wno-trigraphs)
add_definitions(-DGLOBAL_DATA_DIRECTORY="${DATA_DIRECTORY}/")
if (BUILD_TESTS)
option(CATCH_BUILD_STATIC_LIBRARY "" ON)
set(CATCH_BUILD_STATIC_LIBRARY ON)
add_subdirectory(External/Catch2/)
# Pull in catch_discover_tests definition
@@ -293,10 +301,11 @@ if(ENABLE_WERROR OR ENABLE_STRICT_WERROR)
endif()
endif()
set(FEX_TUNE_COMPILE_FLAGS)
if (NOT TUNE_ARCH STREQUAL "generic")
check_cxx_compiler_flag("-march=${TUNE_ARCH}" COMPILER_SUPPORTS_ARCH_TYPE)
if(COMPILER_SUPPORTS_ARCH_TYPE)
add_compile_options("-march=${TUNE_ARCH}")
list(APPEND FEX_TUNE_COMPILE_FLAGS "-march=${TUNE_ARCH}")
else()
message(FATAL_ERROR "Trying to compile arch type '${TUNE_ARCH}' but the compiler doesn't support this")
endif()
@@ -309,7 +318,7 @@ if (TUNE_CPU STREQUAL "native")
# Clang can not currently check for native Apple M1 type in hypervisor. Currently disabled
check_cxx_compiler_flag("-mcpu=native" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
add_compile_options("-mcpu=native")
list(APPEND FEX_TUNE_COMPILE_FLAGS "-mcpu=native")
endif()
else()
# Due to an oversight in llvm, it declares any reasonably new Kryo CPU to only be ARMv8.0
@@ -323,19 +332,19 @@ if (TUNE_CPU STREQUAL "native")
check_cxx_compiler_flag("-mcpu=${AARCH64_CPU}" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
add_compile_options("-mcpu=${AARCH64_CPU}")
list(APPEND FEX_TUNE_COMPILE_FLAGS "-mcpu=${AARCH64_CPU}")
endif()
endif()
else()
check_cxx_compiler_flag("-march=native" COMPILER_SUPPORTS_MARCH_NATIVE)
if(COMPILER_SUPPORTS_MARCH_NATIVE)
add_compile_options("-march=native")
list(APPEND FEX_TUNE_COMPILE_FLAGS "-march=native")
endif()
endif()
else()
check_cxx_compiler_flag("-mcpu=${TUNE_CPU}" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
add_compile_options("-mcpu=${TUNE_CPU}")
list(APPEND FEX_TUNE_COMPILE_FLAGS "-mcpu=${TUNE_CPU}")
else()
message(FATAL_ERROR "Trying to compile cpu type '${TUNE_CPU}' but the compiler doesn't support this")
endif()
@@ -349,57 +358,6 @@ if (ENABLE_IWYU)
endif()
endif()
if (ENABLE_CLANG_FORMAT)
find_program(CLANG_TIDY_EXE "clang-tidy")
if (NOT CLANG_TIDY_EXE)
message(FATAL_ERROR "Couldn't find clang-tidy")
endif()
set(CLANG_TIDY_FLAGS
"-checks=*"
"-fuchsia*"
"-bugprone-macro-parentheses"
"-clang-analyzer-core.*"
"-cppcoreguidelines-pro-type-*"
"-cppcoreguidelines-pro-bounds-array-to-pointer-decay"
"-cppcoreguidelines-pro-bounds-pointer-arithmetic"
"-cppcoreguidelines-avoid-c-arrays"
"-cppcoreguidelines-avoid-magic-numbers"
"-cppcoreguidelines-pro-bounds-constant-array-index"
"-cppcoreguidelines-no-malloc"
"-cppcoreguidelines-special-member-functions"
"-cppcoreguidelines-owning-memory"
"-cppcoreguidelines-macro-usage"
"-cppcoreguidelines-avoid-goto"
"-google-readability-function-size"
"-google-readability-namespace-comments"
"-google-readability-braces-around-statements"
"-google-build-using-namespace"
"-hicpp-*"
"-llvm-namespace-comment"
"-llvm-include-order" # Messes up with case sensitivity
"-llvmlibc-*"
"-misc-unused-parameters"
"-modernize-loop-convert"
"-modernize-use-auto"
"-modernize-avoid-c-arrays"
"-modernize-use-nodiscard"
"readability-*"
"-readability-function-size"
"-readability-implicit-bool-conversion"
"-readability-braces-around-statements"
"-readability-else-after-return"
"-readability-magic-numbers"
"-readability-named-parameter"
"-readability-uppercase-literal-suffix"
"-cert-err34-c"
"-cert-err58-cpp"
"-bugprone-exception-escape"
)
string(REPLACE ";" "," CLANG_TIDY_FLAGS "${CLANG_TIDY_FLAGS}")
set(CMAKE_CXX_CLANG_TIDY ${CLANG_TIDY_EXE} "${CLANG_TIDY_FLAGS}")
endif()
add_compile_options(-Wall)
configure_file(
@@ -410,9 +368,19 @@ if (BUILD_TESTS)
include(CTest)
enable_testing()
message(STATUS "Unit tests are enabled")
set (TEST_JOB_COUNT "" CACHE STRING "Override number of parallel jobs to use while running tests")
if (TEST_JOB_COUNT)
message(STATUS "Running tests with ${TEST_JOB_COUNT} jobs")
endif()
if (CMAKE_VERSION VERSION_LESS "3.29")
execute_process(COMMAND "nproc" OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE TEST_JOB_COUNT)
endif()
set(TEST_JOB_FLAG "-j${TEST_JOB_COUNT}")
endif()
add_subdirectory(FEXHeaderUtils/)
add_subdirectory(CodeEmitter/)
add_subdirectory(FEXCore/)
# Binfmt_misc files must be installed prior to Source/ installs
+2
View File
@@ -0,0 +1,2 @@
add_library(CodeEmitter INTERFACE)
target_include_directories(CodeEmitter INTERFACE .)
File diff suppressed because it is too large. Load diff
@@ -8,23 +8,25 @@ public:
public:
// Conditional branch immediate
///< Branch conditional
void b(FEXCore::ARMEmitter::Condition Cond, uint32_t Imm) {
void b(ARMEmitter::Condition Cond, uint32_t Imm) {
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, Imm);
}
void b(FEXCore::ARMEmitter::Condition Cond, BackwardLabel const* Label) {
void b(ARMEmitter::Condition Cond, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, Imm >> 2);
}
void b(FEXCore::ARMEmitter::Condition Cond, ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::BC });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void b(ARMEmitter::Condition Cond, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, 0);
}
void b(FEXCore::ARMEmitter::Condition Cond, BiDirectionalLabel *Label) {
void b(ARMEmitter::Condition Cond, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
b(Cond, &Label->Backward);
}
@@ -34,24 +36,26 @@ public:
}
///< Branch consistent conditional
void bc(FEXCore::ARMEmitter::Condition Cond, uint32_t Imm) {
void bc(ARMEmitter::Condition Cond, uint32_t Imm) {
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, Imm);
}
void bc(FEXCore::ARMEmitter::Condition Cond, BackwardLabel const* Label) {
void bc(ARMEmitter::Condition Cond, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, Imm >> 2);
}
void bc(FEXCore::ARMEmitter::Condition Cond, ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::BC });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void bc(ARMEmitter::Condition Cond, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, 0);
}
void bc(FEXCore::ARMEmitter::Condition Cond, BiDirectionalLabel *Label) {
void bc(ARMEmitter::Condition Cond, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
bc(Cond, &Label->Backward);
}
@@ -61,7 +65,7 @@ public:
}
// Unconditional branch register
void br(FEXCore::ARMEmitter::Register rn) {
void br(ARMEmitter::Register rn) {
constexpr uint32_t Op = 0b1101011 << 25 |
0b0'000 << 21 | // opc
0b1'1111 << 16 | // op2
@@ -70,7 +74,7 @@ public:
UnconditionalBranch(Op, rn);
}
void blr(FEXCore::ARMEmitter::Register rn) {
void blr(ARMEmitter::Register rn) {
constexpr uint32_t Op = 0b1101011 << 25 |
0b0'001 << 21 | // opc
0b1'1111 << 16 | // op2
@@ -79,7 +83,7 @@ public:
UnconditionalBranch(Op, rn);
}
void ret(FEXCore::ARMEmitter::Register rn = FEXCore::ARMEmitter::Reg::r30) {
void ret(ARMEmitter::Register rn = ARMEmitter::Reg::r30) {
constexpr uint32_t Op = 0b1101011 << 25 |
0b0'010 << 21 | // opc
0b1'1111 << 16 | // op2
@@ -102,8 +106,10 @@ public:
UnconditionalBranch(Op, Imm >> 2);
}
void b(ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::B });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void b(LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::B });
constexpr uint32_t Op = 0b0001'01 << 26;
UnconditionalBranch(Op, 0);
@@ -131,8 +137,10 @@ public:
UnconditionalBranch(Op, Imm >> 2);
}
void bl(ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::B });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void bl(LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::B });
constexpr uint32_t Op = 0b1001'01 << 26;
UnconditionalBranch(Op, 0);
@@ -148,13 +156,13 @@ public:
}
// Compare and branch
void cbz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, uint32_t Imm) {
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, Imm);
}
void cbz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, BackwardLabel const* Label) {
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
@@ -163,15 +171,17 @@ public:
CompareAndBranch(Op, s, rt, Imm >> 2);
}
void cbz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::BC });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, 0);
}
void cbz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, BiDirectionalLabel *Label) {
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
cbz(s, rt, &Label->Backward);
}
@@ -180,13 +190,13 @@ public:
}
}
void cbnz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, uint32_t Imm) {
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, Imm);
}
void cbnz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, BackwardLabel const* Label) {
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
@@ -195,15 +205,17 @@ public:
CompareAndBranch(Op, s, rt, Imm >> 2);
}
void cbnz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::BC });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, 0);
}
void cbnz(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, BiDirectionalLabel *Label) {
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
cbnz(s, rt, &Label->Backward);
}
@@ -213,12 +225,12 @@ public:
}
// Test and branch immediate
void tbz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
void tbz(ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, Imm);
}
void tbz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, BackwardLabel const* Label) {
void tbz(ARMEmitter::Register rt, uint32_t Bit, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
@@ -226,15 +238,18 @@ public:
TestAndBranch(Op, rt, Bit, Imm >> 2);
}
void tbz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::TEST_BRANCH });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void tbz(ARMEmitter::Register rt, uint32_t Bit, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::TEST_BRANCH });
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, 0);
}
void tbz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel *Label) {
void tbz(ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
tbz(rt, Bit, &Label->Backward);
}
@@ -243,12 +258,12 @@ public:
}
}
void tbnz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
void tbnz(ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0111 << 24;
TestAndBranch(Op, rt, Bit, Imm);
}
void tbnz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, BackwardLabel const* Label) {
void tbnz(ARMEmitter::Register rt, uint32_t Bit, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
@@ -256,14 +271,17 @@ public:
TestAndBranch(Op, rt, Bit, Imm >> 2);
}
void tbnz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, ForwardLabel *Label) {
Label->Insts.emplace_back(ForwardLabel::Instructions{ .Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::Instructions::InstType::TEST_BRANCH });
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void tbnz(ARMEmitter::Register rt, uint32_t Bit, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::TEST_BRANCH });
constexpr uint32_t Op = 0b0011'0111 << 24;
TestAndBranch(Op, rt, Bit, 0);
}
void tbnz(FEXCore::ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel *Label) {
void tbnz(ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
tbnz(rt, Bit, &Label->Backward);
}
@@ -274,7 +292,7 @@ public:
private:
// Conditional branch immediate
void Branch_Conditional(uint32_t Op, uint32_t Op1, uint32_t Op0, FEXCore::ARMEmitter::Condition Cond, uint32_t Imm) {
void Branch_Conditional(uint32_t Op, uint32_t Op1, uint32_t Op0, ARMEmitter::Condition Cond, uint32_t Imm) {
uint32_t Instr = Op;
Instr |= Op1 << 24;
@@ -286,7 +304,7 @@ private:
}
// Unconditional branch register
void UnconditionalBranch(uint32_t Op, FEXCore::ARMEmitter::Register rn) {
void UnconditionalBranch(uint32_t Op, ARMEmitter::Register rn) {
uint32_t Instr = Op;
Instr |= Encode_rn(rn);
dc32(Instr);
@@ -300,8 +318,8 @@ private:
}
// Compare and branch
void CompareAndBranch(uint32_t Op, FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register rt, uint32_t Imm) {
const uint32_t SF = s == FEXCore::ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
void CompareAndBranch(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
uint32_t Instr = Op;
@@ -312,7 +330,7 @@ private:
}
// Test and branch - immediate
void TestAndBranch(uint32_t Op, FEXCore::ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
void TestAndBranch(uint32_t Op, ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
uint32_t Instr = Op;
Instr |= (Bit >> 5) << 31;
+106
View File
@@ -0,0 +1,106 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <cstddef>
#include <cstdint>
#include <cstring>
namespace ARMEmitter {
class Buffer {
public:
Buffer() {
SetBuffer(nullptr, 0);
}
Buffer(uint8_t* Base, uint64_t BaseSize) {
SetBuffer(Base, BaseSize);
}
void SetBuffer(uint8_t* Base, uint64_t BaseSize) {
BufferBase = Base;
CurrentOffset = BufferBase;
Size = BaseSize;
}
void dc8(uint8_t Data) {
decltype(Data)* Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void dc16(uint16_t Data) {
decltype(Data)* Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void dc32(uint32_t Data) {
decltype(Data)* Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void dc64(uint64_t Data) {
decltype(Data)* Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void EmitString(const char* String) {
const auto StringLength = strlen(String);
memcpy(CurrentOffset, String, StringLength);
CurrentOffset += StringLength;
}
void Align() {
// Align the buffer to instruction size
auto CurrentAlignment = reinterpret_cast<uint64_t>(CurrentOffset) & 0b11;
if (!CurrentAlignment) {
return;
}
CurrentOffset += 4 - CurrentAlignment;
}
template<typename T>
T GetCursorAddress() const {
return reinterpret_cast<T>(CurrentOffset);
}
static void ClearICache(void* Begin, std::size_t Length) {
__builtin___clear_cache(static_cast<char*>(Begin), static_cast<char*>(Begin) + Length);
}
size_t GetCursorOffset() const {
return static_cast<size_t>(CurrentOffset - BufferBase);
}
uint8_t* GetBufferBase() const {
return BufferBase;
}
void CursorIncrement(size_t Size) {
CurrentOffset += Size;
}
void SetCursorOffset(size_t Offset) {
CurrentOffset = BufferBase + Offset;
}
uint64_t GetBufferSize() const {
return Size;
}
template<typename T>
size_t GetCursorOffsetFromAddress(const T* Address) const {
return static_cast<size_t>(reinterpret_cast<const uint8_t*>(Address) - BufferBase);
}
protected:
void ResetBuffer() {
CurrentOffset = BufferBase;
}
uint8_t* BufferBase;
uint8_t* CurrentOffset;
uint64_t Size;
};
} // namespace ARMEmitter
+831
View File
@@ -0,0 +1,831 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <FEXCore/Utils/CompilerDefs.h>
#include <FEXCore/Utils/EnumUtils.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/MathUtils.h>
#include <FEXCore/fextl/vector.h>
#include <FEXHeaderUtils/BitUtils.h>
#include <CodeEmitter/Buffer.h>
#include <CodeEmitter/Registers.h>
#include <array>
#include <cstdint>
#include <utility>
#include <type_traits>
/*
* Welcome to FEX-Emu's custom AArch64 emitter.
* This was written specifically to avoid the performance cost of the vixl emitter.
*
* There are some specific design constraints in this design to target a couple features:
* - High performance
* - Low CPU cache performance hit
* - Significantly reduced code footprint
* - Low number of branches
*
* These requirements are mostly achieved by removing a bunch of developer conveniences
* that vixl provides. The developer needs to take a lot of care to not shoot themselves in the foot.
*
* Misc design decisions:
* - Registers are encoded as basic uint32_t enums.
* - Converting between different registers is zero-cost.
* - Passing around as arguments are as cheap as registers
* - Contrast to vixl where every register requires living on the stack.
* - Registers can get encoded in to instructions with a simple `BFM` instruction.
*
* - Instructions are very simply emitted, allowing direct inlining most of the time.
* - These are simple enough that multiple back-to-back instructions get optimized to 128-bit load-store operations.
* - Contrast to vixl where pretty much no instruction emitter gets inlined.
*
* - Instruction emitters are /mostly/ unsized. Most instructions take a size argument first, which gets encoded
* directly in to the instruction.
* - Contrast to vixl where the register arguments are how the instructions determine operating size.
* - Size argument allows FEX to use `CSEL` to select a size at runtime, instead of branching.
* - Some instructions are explicitly sized based on register type. Read comments in the respective `inl` files to
* see why.
* Some scalar/vector operations are an example of this.
*
* - Almost zero helper functions.
* - Primary exception to this rule is load-store operations. These will use a helper to make
* it easier to select the correct load-store instruction. Mostly because these are a nightmare selecting
* the right instruction.
*/
namespace ARMEmitter {
/*
* This `Size` enum is used for most ALU operations.
* These follow the AArch64 encoding style in most cases.
*/
enum class Size : uint32_t {
i32Bit = 0,
i64Bit,
};
// This allows us to get the `Size` enum in bits.
[[nodiscard]]
constexpr size_t RegSizeInBits(Size size) {
return size_t {32} << FEXCore::ToUnderlying(size);
}
/* This `SubRegSize` enum is used for most ASIMD operations.
* These follow the AArch64 encoding style in most cases.
*/
enum class SubRegSize : uint32_t {
i8Bit = 0b00,
i16Bit = 0b01,
i32Bit = 0b10,
i64Bit = 0b11,
i128Bit = 0b100,
};
// This allows us to get the `SubRegSize` in bits.
[[nodiscard]]
constexpr size_t SubRegSizeInBits(SubRegSize size) {
return size_t {8} << FEXCore::ToUnderlying(size);
}
/* This `ScalarRegSize` enum is used for most scalar float
* operations.
*
* This is specifically duplicated from `SubRegSize` to have strongly
* typed functions.
*
* `ScalarRegSize` specifically doesn't have `i128Bit` because scalar operations
* can't operate at 128-bit.
*/
enum class ScalarRegSize : uint32_t {
i8Bit = 0b00,
i16Bit = 0b01,
i32Bit = 0b10,
i64Bit = 0b11,
};
// This allows us to get the `ScalarRegSize` in bits.
[[nodiscard]]
constexpr size_t ScalarRegSizeInBits(ScalarRegSize size) {
return size_t {8} << FEXCore::ToUnderlying(size);
}
/* This `VectorRegSizePair` union allows us to have an overlapping type
* to select a scalar operation or a vector depending on which operation
* we pass in.
* Useful in FEX's vector operations that behave as scalar or vector
* depending on various factors. But since the operation will have the sa,e
* element size, we want to choose the operation more easily
*/
union VectorRegSizePair {
ScalarRegSize Scalar;
SubRegSize Vector;
};
// This allows us to create a `VectorRegSizePair` union.
[[nodiscard]]
constexpr VectorRegSizePair ToVectorSizePair(SubRegSize size) {
return VectorRegSizePair {.Vector = size};
}
[[nodiscard]]
constexpr VectorRegSizePair ToVectorSizePair(ScalarRegSize size) {
return VectorRegSizePair {.Scalar = size};
}
// This `ShiftType` enum is used for ALU shift-register encoded instructions.
enum class ShiftType : uint32_t {
LSL = 0,
LSR,
ASR,
ROR,
};
// This `ExtendedType` enum is used for ALU extended-register encoded instructions.
enum class ExtendedType : uint32_t {
UXTB = 0b000,
UXTH = 0b001,
UXTW = 0b010,
UXTX = 0b011,
SXTB = 0b100,
SXTH = 0b101,
SXTW = 0b110,
SXTX = 0b111,
LSL_32 = UXTW,
LSL_64 = UXTX,
};
// This `Condition` enum is used for various conditional instructions.
enum class Condition : uint32_t {
// Meaning: Int - Float
CC_EQ = 0, // Equal - Equal
CC_NE, // Not Eq - Not Eq or unordered
CC_CS, // Carry set - Greater than, equal, or unordered
CC_CC, // Carry clear - Less than
CC_MI, // Minus/Negative - Less than
CC_PL, // Plus, positive or zero - GT, equal, or unordered
CC_VS, // Overflow - Unordered
CC_VC, // No Overflow - Ordered
CC_HI, // Unsigned higher - GT, or unordered
CC_LS, // Unsigned lower or same - LT or EQ
CC_GE, // Signed GT or EQ - GT or EQ
CC_LT, // Signed LT - LT or Unordered
CC_GT, // Signed GT - GT
CC_LE, // Signed LT or EQ - LT, EQ, or Unordered
CC_AL, // Always - Always
CC_NV, // Always - Always
// Aliases
CC_HS = CC_CS,
CC_LO = CC_CC,
};
/*
* This `StatusFlags` enum is used for conditional compare encoded instructions.
* These directly encode to the `nzcv` flags.
*/
enum class StatusFlags : uint32_t {
None = 0,
Flag_V = 0b0001,
Flag_C = 0b0010,
Flag_Z = 0b0100,
Flag_N = 0b1000,
Flag_NZCV = Flag_N | Flag_Z | Flag_C | Flag_V,
};
/*
* This `IndexType` enum is used for load-store instructions.
* Not all load-store instructions use this, so the user needs to be careful.
*/
enum class IndexType {
POST,
OFFSET,
PRE,
UNPRIVILEGED,
};
// Used with adr and scalar + vector load/store variants to denote
// a modifier operation.
enum class SVEModType : uint8_t {
MOD_UXTW,
MOD_SXTW,
MOD_LSL,
MOD_NONE,
};
/* This `SVEMemOperand` class is used for the helper SVE load-store instructions.
* Load-store instructions are quite expressive, so having a helper that handles these differences is worth it.
*/
class SVEMemOperand final {
public:
enum class Type {
ScalarPlusScalar,
ScalarPlusImm,
ScalarPlusVector,
VectorPlusImm,
};
SVEMemOperand(XRegister rn, XRegister rm = XReg::zr)
: rn {rn}
, MemType {Type::ScalarPlusScalar}
, MetaType {.ScalarScalarType {
.rm = rm,
}} {}
SVEMemOperand(XRegister rn, int32_t imm = 0)
: rn {rn}
, MemType {Type::ScalarPlusImm}
, MetaType {.ScalarImmType {
.Imm = imm,
}} {}
SVEMemOperand(XRegister rn, ZRegister zm, SVEModType mod = SVEModType::MOD_NONE, uint8_t scale = 0)
: rn {rn}
, MemType {Type::ScalarPlusVector}
, MetaType {.ScalarVectorType {
.zm = zm,
.mod = mod,
.scale = scale,
}} {}
SVEMemOperand(ZRegister zn, uint32_t imm)
: rn {Register {zn.Idx()}}
, MemType {Type::VectorPlusImm}
, MetaType {.VectorImmType {
.Imm = imm,
}} {}
[[nodiscard]]
bool IsScalarPlusScalar() const {
return MemType == Type::ScalarPlusScalar;
}
[[nodiscard]]
bool IsScalarPlusImm() const {
return MemType == Type::ScalarPlusImm;
}
[[nodiscard]]
bool IsScalarPlusVector() const {
return MemType == Type::ScalarPlusVector;
}
[[nodiscard]]
bool IsVectorPlusImm() const {
return MemType == Type::VectorPlusImm;
}
union Data {
struct {
Register rm;
} ScalarScalarType;
struct {
int32_t Imm;
} ScalarImmType;
struct {
ZRegister zm;
SVEModType mod;
uint8_t scale;
} ScalarVectorType;
struct {
// rn will be a ZRegister
uint32_t Imm;
} VectorImmType;
};
Register rn;
Type MemType;
Data MetaType;
};
/* This `ExtendedMemOperand` class is used for the helper load-store instructions.
* Load-store instructions are quite expressive, so having a helper that handles these differences is worth it.
*/
class ExtendedMemOperand final {
public:
ExtendedMemOperand(XRegister rn, XRegister rm = XReg::zr, ExtendedType Option = ExtendedType::LSL_64, uint32_t Shift = 0)
: rn {rn}
, MetaType {.ExtendedType {
.Header = {.MemType = TYPE_EXTENDED},
.rm = rm,
.Option = Option,
.Shift = Shift,
}} {}
ExtendedMemOperand(XRegister rn, IndexType Index = IndexType::OFFSET, int32_t Imm = 0)
: rn {rn}
, MetaType {.ImmType {
.Header = {.MemType = TYPE_IMM},
.Index = Index,
.Imm = Imm,
}} {}
Register rn;
enum Type {
TYPE_EXTENDED,
TYPE_IMM,
};
struct HeaderStruct {
Type MemType;
};
union {
HeaderStruct Header;
struct {
HeaderStruct Header;
Register rm;
ExtendedType Option;
uint32_t Shift;
} ExtendedType;
struct {
HeaderStruct Header;
IndexType Index;
int32_t Imm;
} ImmType;
} MetaType;
};
template<uint32_t op0, uint32_t op1, uint32_t CRn, uint32_t CRm, uint32_t op2>
constexpr uint32_t GenSystemReg() {
return op0 << 19 | op1 << 16 | CRn << 12 | CRm << 8 | op2 << 5;
};
// This `SystemRegister` enum is used for the mrs/msr instructions.
enum class SystemRegister : uint32_t {
CTR_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b001>(),
DCZID_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b111>(),
TPIDR_EL0 = GenSystemReg<0b11, 0b011, 0b1101, 0b0000, 0b010>(),
RNDR = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b000>(),
RNDRRS = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b001>(),
NZCV = GenSystemReg<0b11, 0b011, 0b0100, 0b0010, 0b000>(),
FPCR = GenSystemReg<0b11, 0b011, 0b0100, 0b0100, 0b000>(),
CNTFRQ_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b000>(),
CNTVCT_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b010>(),
};
template<uint32_t op1, uint32_t CRm, uint32_t op2>
constexpr uint32_t GenDCReg() {
return op1 << 16 | CRm << 8 | op2 << 5;
};
// This `DataCacheOperation` enum is used for the dc instruction.
enum class DataCacheOperation : uint32_t {
IVAC = GenDCReg<0b000, 0b0110, 0b001>(),
ISW = GenDCReg<0b000, 0b0110, 0b010>(),
CSW = GenDCReg<0b000, 0b1010, 0b010>(),
CISW = GenDCReg<0b000, 0b1110, 0b010>(),
ZVA = GenDCReg<0b011, 0b0100, 0b001>(),
CVAC = GenDCReg<0b011, 0b1010, 0b001>(),
CVAU = GenDCReg<0b011, 0b1011, 0b001>(),
CIVAC = GenDCReg<0b011, 0b1110, 0b001>(),
// MTE2
IGVAC = GenDCReg<0b000, 0b0110, 0b011>(),
IGSW = GenDCReg<0b000, 0b0110, 0b100>(),
IGDVAC = GenDCReg<0b000, 0b0110, 0b101>(),
IGDSW = GenDCReg<0b000, 0b0110, 0b110>(),
CGSW = GenDCReg<0b000, 0b1010, 0b100>(),
CGDSW = GenDCReg<0b000, 0b1010, 0b110>(),
CIGSW = GenDCReg<0b000, 0b1110, 0b100>(),
CIGDSW = GenDCReg<0b000, 0b1110, 0b110>(),
// MTE
GVA = GenDCReg<0b011, 0b0100, 0b011>(),
GZVA = GenDCReg<0b011, 0b0100, 0b100>(),
CGVAC = GenDCReg<0b011, 0b1010, 0b011>(),
CGDVAC = GenDCReg<0b011, 0b1010, 0b101>(),
CGVAP = GenDCReg<0b011, 0b1100, 0b011>(),
CGDVAP = GenDCReg<0b011, 0b1100, 0b101>(),
CGVADP = GenDCReg<0b011, 0b1101, 0b011>(),
CGDVADP = GenDCReg<0b011, 0b1101, 0b101>(),
CIGVAC = GenDCReg<0b011, 0b1110, 0b011>(),
CIGDVAC = GenDCReg<0b011, 0b1110, 0b101>(),
// DPB
CVAP = GenDCReg<0b011, 0b1100, 0b001>(),
// DPB2
CVADP = GenDCReg<0b011, 0b1101, 0b001>(),
};
template<uint32_t CRm, uint32_t op2>
constexpr uint32_t GenHintBarrierReg() {
return CRm << 8 | op2 << 5;
}
// This `HintRegister` enum is used for the hint instruction.
enum class HintRegister : uint32_t {
NOP = GenHintBarrierReg<0b0000, 0b000>(),
YIELD = GenHintBarrierReg<0b0000, 0b001>(),
WFE = GenHintBarrierReg<0b0000, 0b010>(),
WFI = GenHintBarrierReg<0b0000, 0b011>(),
SEV = GenHintBarrierReg<0b0000, 0b100>(),
SEVL = GenHintBarrierReg<0b0000, 0b101>(),
DGH = GenHintBarrierReg<0b0000, 0b110>(),
CSDB = GenHintBarrierReg<0b0010, 0b100>(),
};
// This `BarrierRegister` enum is used for the various barrier instructions.
enum class BarrierRegister : uint32_t {
CLREX = GenHintBarrierReg<0b0000, 0b010>(),
TCOMMIT = GenHintBarrierReg<0b0000, 0b011>(),
DSB = GenHintBarrierReg<0b0000, 0b100>(),
DMB = GenHintBarrierReg<0b0000, 0b101>(),
ISB = GenHintBarrierReg<0b0000, 0b110>(),
SB = GenHintBarrierReg<0b0000, 0b111>(),
};
// This `BarrierScope` enum is used for the dsb/dmb instructions.
enum class BarrierScope : uint32_t {
// Outer shareable
OSHLD = 0b0001,
OSHST = 0b0010,
OSH = 0b0011,
// Non shareable
NSHLD = 0b0101,
NSHST = 0b0110,
NSH = 0b0111,
// Inner shareable
ISHLD = 0b1001,
ISHST = 0b1010,
ISH = 0b1011,
// Full System visibility
LD = 0b1101,
ST = 0b1110,
SY = 0b1111,
};
// This `Prefetch` enum is used for prefetch instructions.
enum class Prefetch : uint32_t {
// Prefetch for load
PLDL1KEEP = 0b00000,
PLDL1STRM = 0b00001,
PLDL2KEEP = 0b00010,
PLDL2STRM = 0b00011,
PLDL3KEEP = 0b00100,
PLDL3STRM = 0b00101,
// Preload instructions
PLIL1KEEP = 0b01000,
PLIL1STRM = 0b01001,
PLIL2KEEP = 0b01010,
PLIL2STRM = 0b01011,
PLIL3KEEP = 0b01100,
PLIL3STRM = 0b01101,
// Preload for store
PSTL1KEEP = 0b10000,
PSTL1STRM = 0b10001,
PSTL2KEEP = 0b10010,
PSTL2STRM = 0b10011,
PSTL3KEEP = 0b10100,
PSTL3STRM = 0b10101,
};
// This `PredicatePattern` enun is used for some SVE instructions.
enum class PredicatePattern : uint32_t {
SVE_POW2 = 0b00000,
SVE_VL1 = 0b00001,
SVE_VL2 = 0b00010,
SVE_VL3 = 0b00011,
SVE_VL4 = 0b00100,
SVE_VL5 = 0b00101,
SVE_VL6 = 0b00110,
SVE_VL7 = 0b00111,
SVE_VL8 = 0b01000,
SVE_VL16 = 0b01001,
SVE_VL32 = 0b01010,
SVE_VL64 = 0b01011,
SVE_VL128 = 0b01100,
SVE_VL256 = 0b01101,
SVE_MUL4 = 0b11101,
SVE_MUL3 = 0b11110,
SVE_ALL = 0b11111,
};
// Used with SVE FP immediate arithmetic instructions
enum class SVEFAddSubImm : uint32_t {
_0_5,
_1_0,
};
enum class SVEFMulImm : uint32_t {
_0_5,
_2_0,
};
enum class SVEFMaxMinImm : uint32_t {
_0_0,
_1_0,
};
/* This `BackwardLabel` struct used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is logically `below` an instruction that uses it.
* Which means that a branch would jump backwards.
*/
struct BackwardLabel {
uint8_t* Location {};
};
/* This `SingleUseForwardLabel` struct used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is logically `above` an instruction that uses it.
* Which means that a branch would jump forwards.
*
* The `ForwardLabel` struct can be bound to multiple instructions, so it needs a vector for each bind instruction type.
*/
struct SingleUseForwardLabel {
enum class InstType {
UNKNOWN,
ADR,
ADRP,
B,
BC,
TEST_BRANCH,
RELATIVE_LOAD,
LONG_ADDRESS_GEN,
};
uint8_t* Location {};
InstType Type = InstType::UNKNOWN;
};
struct ForwardLabel {
fextl::vector<SingleUseForwardLabel> Insts {};
};
/* This `BiDirectionalLabel` struct used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is in either direction of an instruction that uses it.
* Which means a branch could jump backwards or forwards depending on situation.
*/
struct BiDirectionalLabel {
BackwardLabel Backward;
ForwardLabel Forward;
};
static inline void AddLocationToLabel(SingleUseForwardLabel* Label, SingleUseForwardLabel&& Location) {
LOGMAN_THROW_A_FMT(Label->Type == SingleUseForwardLabel::InstType::UNKNOWN, "Trying to bind a SingleUseForwardLabel to multiple "
"locations. Use ForwardLabel instead.");
*Label = std::move(Location);
}
static inline void AddLocationToLabel(ForwardLabel* Label, SingleUseForwardLabel&& Location) {
Label->Insts.emplace_back(std::move(Location));
}
// Some FCMA ASIMD instructions support a rotation argument.
enum class Rotation : uint32_t {
ROTATE_0 = 0b00,
ROTATE_90 = 0b01,
ROTATE_180 = 0b10,
ROTATE_270 = 0b11,
};
// Concept for contraining some instructions to accept only an XRegister or WRegister.
// Particularly for operations that differ encodings depending on which one is used.
template<typename T>
concept IsXOrWRegister = std::is_same_v<T, XRegister> || std::is_same_v<T, WRegister>;
// Whether or not a given set of vector registers are sequential
// in increasing order as far as the register file is concerned (modulo its size)
//
// For example, a set of registers like:
//
// v1, v2, v3 and
// v31, v0, v1
//
// would both be considered sequential sequences, and some instructions in particular
// limit register lists to these kind of sequences.
//
template<typename T, typename... Args>
constexpr bool AreVectorsSequential(T first, const Args&... args) {
// Ensure we always have a pair of registers to compare against.
static_assert(sizeof...(args) >= 1, "Number of arguments must be greater than 1");
const auto fn = [](auto& lhs, const auto& rhs) {
const auto result = ((lhs.Idx() + 1) % 32) == rhs.Idx();
lhs = rhs;
return result;
};
return (fn(first, args) && ...);
}
// This is an emitter that is designed around the smallest code bloat as possible.
// Eschewing most developer convenience in order to keep code as small as possible.
// Choices:
// - Size of ops passed as an argument rather than template to let the compiler use csel instead of branching.
// - Registers are unsized so they can be passed in a GPR and not need conversion operations
class Emitter : public ARMEmitter::Buffer {
public:
Emitter() = default;
Emitter(uint8_t* Base, uint64_t BaseSize)
: Buffer(Base, BaseSize) {}
// Bind a backward label to an address.
// Address that is bound is the current emitter location.
void Bind(BackwardLabel* Label) {
LOGMAN_THROW_AA_FMT(Label->Location == nullptr, "Trying to bind a label twice");
Label->Location = GetCursorAddress<uint8_t*>();
}
void Bind(const SingleUseForwardLabel* Label) {
uint8_t* CurrentAddress = GetCursorAddress<uint8_t*>();
// Patch up the instructions
switch (Label->Type) {
case SingleUseForwardLabel::InstType::ADR: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(IsADRRange(Imm), "Unscaled offset too large");
uint32_t InstMask = 0b11 << 29 | 0b1111'1111'1111'1111'111 << 5;
uint32_t Offset = static_cast<uint32_t>(Imm) & 0x3F'FFFF;
uint32_t Inst = *Instruction & ~InstMask;
Inst |= (Offset & 0b11) << 29;
Inst |= (Offset >> 2) << 5;
*Instruction = Inst;
break;
}
case SingleUseForwardLabel::InstType::ADRP: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(IsADRPRange(Imm) && IsADRPAligned(Imm), "Unscaled offset too large");
Imm >>= 12;
uint32_t InstMask = 0b11 << 29 | 0b1111'1111'1111'1111'111 << 5;
uint32_t Offset = static_cast<uint32_t>(Imm) & 0x3F'FFFF;
uint32_t Inst = *Instruction & ~InstMask;
Inst |= (Offset & 0b11) << 29;
Inst |= (Offset >> 2) << 5;
*Instruction = Inst;
break;
}
case SingleUseForwardLabel::InstType::B: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -134217728 && Imm <= 134217724 && ((Imm & 0b11) == 0), "Unscaled offset too large");
Imm >>= 2;
uint32_t InstMask = 0x3FF'FFFF;
uint32_t Offset = static_cast<uint32_t>(Imm) & InstMask;
uint32_t Inst = *Instruction & ~InstMask;
Inst |= Offset;
*Instruction = Inst;
break;
}
case SingleUseForwardLabel::InstType::TEST_BRANCH: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
Imm >>= 2;
uint32_t InstMask = 0x3FFF;
uint32_t Offset = static_cast<uint32_t>(Imm) & InstMask;
uint32_t Inst = *Instruction & ~(InstMask << 5);
Inst |= Offset << 5;
*Instruction = Inst;
break;
}
case SingleUseForwardLabel::InstType::BC:
case SingleUseForwardLabel::InstType::RELATIVE_LOAD: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
Imm >>= 2;
uint32_t InstMask = 0x7'FFFF;
uint32_t Offset = static_cast<uint32_t>(Imm) & InstMask;
uint32_t Inst = *Instruction & ~(InstMask << 5);
Inst |= Offset << 5;
*Instruction = Inst;
break;
}
case SingleUseForwardLabel::InstType::LONG_ADDRESS_GEN: {
uint32_t* Instructions = reinterpret_cast<uint32_t*>(Label->Location);
int64_t ImmInstOne = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(&Instructions[0]);
int64_t ImmInstTwo = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(&Instructions[1]);
auto OriginalOffset = GetCursorOffset();
auto InstOffset = GetCursorOffsetFromAddress(Instructions);
SetCursorOffset(InstOffset);
// We encoded the destination register in to the first instruction space.
// Read it back.
ARMEmitter::Register DestReg(Instructions[0]);
if (IsADRRange(ImmInstTwo)) {
// If within ADR range from the second instruction, then we can emit NOP+ADR
nop();
adr(DestReg, static_cast<uint32_t>(ImmInstTwo) & 0x7FFF);
} else if (IsADRPRange(ImmInstOne)) {
// If within ADRP range from the first instruction, then we are /definitely/ in range for the second instruction.
// First check if we are in non-offset range for second instruction.
if (IsADRPAligned(reinterpret_cast<uint64_t>(CurrentAddress))) {
// We can emit nop + adrp
nop();
adrp(DestReg, static_cast<uint32_t>(ImmInstTwo >> 12) & 0x7FFF);
} else {
// Not aligned, need adrp + add
adrp(DestReg, static_cast<uint32_t>(ImmInstOne >> 12) & 0x7FFF);
add(ARMEmitter::Size::i64Bit, DestReg, DestReg, ImmInstOne & 0xFFF);
}
} else {
LOGMAN_MSG_A_FMT("Unscaled offset is too large");
FEX_UNREACHABLE;
}
SetCursorOffset(OriginalOffset);
break;
}
default: LOGMAN_MSG_A_FMT("Unexpected inst type in label fixup");
}
}
// Bind a forward label to a location.
// This walks all the instructions in the label's vector.
// Then backpatching all instructions that have used the label.
template<bool WarnAboutEmpty = false>
void Bind(ForwardLabel* Label) {
if constexpr (WarnAboutEmpty) {
LOGMAN_THROW_A_FMT(Label->Insts.empty() == false, "Binding forward label that didn't have any instructions using it");
}
for (auto& Inst : Label->Insts) {
Bind(&Inst);
}
}
// Bind a bidirectional location to a location.
// Binds both forwards and backwards depending on how the label was used.
void Bind(BiDirectionalLabel* Label) {
if (!Label->Backward.Location) {
Bind(&Label->Backward);
}
Bind<false>(&Label->Forward);
}
#include <CodeEmitter/VixlUtils.inl>
public:
// TODO: Implement SME when it matters.
#include <CodeEmitter/ALUOps.inl>
#include <CodeEmitter/BranchOps.inl>
#include <CodeEmitter/LoadstoreOps.inl>
#include <CodeEmitter/SystemOps.inl>
#include <CodeEmitter/ScalarOps.inl>
#include <CodeEmitter/ASIMDOps.inl>
#include <CodeEmitter/SVEOps.inl>
private:
template<typename T>
uint32_t Encode_ra(T Reg) const {
return Reg.Idx() << 10;
}
uint32_t Encode_ra(uint32_t Reg) const {
return Reg << 10;
}
template<typename T>
uint32_t Encode_rt2(T Reg) const {
return Reg.Idx() << 10;
}
template<>
uint32_t Encode_rt2(uint32_t Reg) const {
return Reg << 10;
}
template<typename T>
uint32_t Encode_rm(T Reg) const {
return Reg.Idx() << 16;
}
uint32_t Encode_rm(uint32_t Reg) const {
return Reg << 16;
}
template<typename T>
uint32_t Encode_rs(T Reg) const {
return Reg.Idx() << 16;
}
uint32_t Encode_rs(uint32_t Reg) const {
return Reg << 16;
}
template<typename T>
uint32_t Encode_rn(T Reg) const {
return Reg.Idx() << 5;
}
uint32_t Encode_rn(uint32_t Reg) const {
return Reg << 5;
}
template<typename T>
uint32_t Encode_rd(T Reg) const {
return Reg.Idx();
}
uint32_t Encode_rd(uint32_t Reg) const {
return Reg;
}
template<typename T>
uint32_t Encode_rt(T Reg) const {
return Reg.Idx();
}
template<>
uint32_t Encode_rt(Prefetch Reg) const {
return FEXCore::ToUnderlying(Reg);
}
uint32_t Encode_rt(uint32_t Reg) const {
return Reg;
}
template<typename T>
uint32_t Encode_pd(T Reg) const {
return FEXCore::ToUnderlying(Reg);
}
};
} // namespace ARMEmitter
File diff suppressed because it is too large. Load diff
@@ -1384,12 +1384,10 @@ public:
}
// SVE predicate initialize
template <SubRegSize size>
void ptrue(PRegister pd, PredicatePattern pattern) {
void ptrue(SubRegSize size, PRegister pd, PredicatePattern pattern) {
SVEPredicateMisc(0b1000, 0b10000, FEXCore::ToUnderlying(pattern), size, pd);
}
template <SubRegSize size>
void ptrues(PRegister pd, PredicatePattern pattern) {
void ptrues(SubRegSize size, PRegister pd, PredicatePattern pattern) {
SVEPredicateMisc(0b1001, 0b10000, FEXCore::ToUnderlying(pattern), size, pd);
}
@@ -1508,13 +1506,14 @@ public:
}
// SVE broadcast floating-point immediate (unpredicated)
void fdup(FEXCore::ARMEmitter::SubRegSize size, FEXCore::ARMEmitter::ZRegister zd, float Value) {
LOGMAN_THROW_AA_FMT(size == FEXCore::ARMEmitter::SubRegSize::i16Bit ||
size == FEXCore::ARMEmitter::SubRegSize::i32Bit ||
size == FEXCore::ARMEmitter::SubRegSize::i64Bit, "Unsupported fmov size");
void fdup(ARMEmitter::SubRegSize size, ARMEmitter::ZRegister zd, float Value) {
LOGMAN_THROW_AA_FMT(size == ARMEmitter::SubRegSize::i16Bit ||
size == ARMEmitter::SubRegSize::i32Bit ||
size == ARMEmitter::SubRegSize::i64Bit, "Unsupported fmov size");
uint32_t Imm{};
if (size == SubRegSize::i16Bit) {
Imm = FP16ToImm8(vixl::Float16(Value));
LOGMAN_MSG_A_FMT("Unsupported");
FEX_UNREACHABLE;
} else if (size == SubRegSize::i32Bit) {
Imm = FP32ToImm8(Value);
} else if (size == SubRegSize::i64Bit) {
@@ -1523,7 +1522,7 @@ public:
SVEBroadcastFloatImmUnpredicated(0b00, 0, Imm, size, zd);
}
void fmov(FEXCore::ARMEmitter::SubRegSize size, FEXCore::ARMEmitter::ZRegister zd, float Value) {
void fmov(ARMEmitter::SubRegSize size, ARMEmitter::ZRegister zd, float Value) {
fdup(size, zd, Value);
}
@@ -3322,7 +3321,18 @@ public:
// SVE Memory - Contiguous Store with Immediate Offset
// SVE contiguous non-temporal store (scalar plus immediate)
// XXX:
void stnt1b(ZRegister zt, PRegister pg, Register rn, int32_t Imm = 0) {
SVEContiguousNontemporalStore(0b00, zt, pg, rn, Imm);
}
void stnt1h(ZRegister zt, PRegister pg, Register rn, int32_t Imm = 0) {
SVEContiguousNontemporalStore(0b01, zt, pg, rn, Imm);
}
void stnt1w(ZRegister zt, PRegister pg, Register rn, int32_t Imm = 0) {
SVEContiguousNontemporalStore(0b10, zt, pg, rn, Imm);
}
void stnt1d(ZRegister zt, PRegister pg, Register rn, int32_t Imm = 0) {
SVEContiguousNontemporalStore(0b11, zt, pg, rn, Imm);
}
// SVE store multiple structures (scalar plus immediate)
void st2b(ZRegister zt1, ZRegister zt2, PRegister pg, Register rn, int32_t Imm = 0) {
@@ -3515,7 +3525,8 @@ private:
size == SubRegSize::i64Bit, "Unsupported fcpy/fmov size");
uint32_t imm{};
if (size == SubRegSize::i16Bit) {
imm = FP16ToImm8(vixl::Float16(value));
LOGMAN_MSG_A_FMT("Unsupported");
FEX_UNREACHABLE;
} else if (size == SubRegSize::i32Bit) {
imm = FP32ToImm8(value);
} else if (size == SubRegSize::i64Bit) {
@@ -3719,7 +3730,7 @@ private:
// SVE bitwise logical operations (predicated)
void SVEBitwiseLogicalPredicated(uint32_t opc, SubRegSize size, PRegister pg, ZRegister zdn, ZRegister zm, ZRegister zd) {
LOGMAN_THROW_AA_FMT(size != FEXCore::ARMEmitter::SubRegSize::i128Bit, "Can't use 128-bit size");
LOGMAN_THROW_AA_FMT(size != ARMEmitter::SubRegSize::i128Bit, "Can't use 128-bit size");
LOGMAN_THROW_A_FMT(zd == zdn, "zd needs to equal zdn");
LOGMAN_THROW_A_FMT(pg <= PReg::p7, "Can only use p0-p7 as a governing predicate");
@@ -4481,6 +4492,22 @@ private:
dc32(Instr);
}
// SVE contiguous non-temporal store (scalar plus immediate)
void SVEContiguousNontemporalStore(uint32_t msz, ZRegister zt, PRegister pg, Register rn, int32_t imm) {
LOGMAN_THROW_A_FMT(pg <= PReg::p7, "Can only use p0-p7 as a governing predicate");
LOGMAN_THROW_AA_FMT(imm >= -8 && imm <= 7,
"Invalid loadstore offset ({}). Must be between [-8, 7]", imm);
const auto imm4 = static_cast<uint32_t>(imm) & 0xF;
uint32_t Instr = 0b1110'0100'0001'0000'1110'0000'0000'0000;
Instr |= msz << 23;
Instr |= imm4 << 16;
Instr |= pg.Idx() << 10;
Instr |= Encode_rn(rn);
Instr |= zt.Idx();
dc32(Instr);
}
void SVEContiguousLoadImm(bool is_store, uint32_t dtype, int32_t imm, PRegister pg, Register rn, ZRegister zt) {
LOGMAN_THROW_A_FMT(pg <= PReg::p7, "Can only use p0-p7 as a governing predicate");
LOGMAN_THROW_AA_FMT(imm >= -8 && imm <= 7,
@@ -4745,7 +4772,7 @@ private:
dc32(Instr);
}
void SVEPermuteVector(uint32_t op0, FEXCore::ARMEmitter::ZRegister zd, FEXCore::ARMEmitter::ZRegister zm, uint32_t Imm) {
void SVEPermuteVector(uint32_t op0, ARMEmitter::ZRegister zd, ARMEmitter::ZRegister zm, uint32_t Imm) {
constexpr uint32_t Op = 0b0000'0101'0010'0000'000 << 13;
uint32_t Instr = Op;
@@ -5230,15 +5257,14 @@ private:
// Alias that returns the equivalently sized unsigned type for a floating-point type T.
template <typename T>
requires(std::is_same_v<T, float> || std::is_same_v<T, double> || std::is_same_v<T, vixl::Float16>)
using FloatToEquivalentUInt = std::conditional_t<std::is_same_v<T, vixl::Float16>, uint16_t,
std::conditional_t<std::is_same_v<T, float>, uint32_t, uint64_t>>;
requires(std::is_same_v<T, float> || std::is_same_v<T, double>)
using FloatToEquivalentUInt = std::conditional_t<std::is_same_v<T, float>, uint32_t, uint64_t>;
// Determines if a floating-point value is capable of being converted
// into an 8-bit immediate. See pseudocode definition of VFPExpandImm
// in ARM A-profile reference manual for a general overview of how this was derived.
template <typename T>
requires(std::is_same_v<T, float> || std::is_same_v<T, double> || std::is_same_v<T, vixl::Float16>)
requires(std::is_same_v<T, float> || std::is_same_v<T, double>)
[[nodiscard, maybe_unused]] static bool IsValidFPValueForImm8(T value) {
const uint64_t bits = FEXCore::BitCast<FloatToEquivalentUInt<T>>(value);
const uint64_t datasize_idx = FEXCore::ilog2(sizeof(T)) - 1;
@@ -5279,18 +5305,6 @@ private:
return true;
}
static uint32_t FP16ToImm8(vixl::Float16 value) {
LOGMAN_THROW_A_FMT(IsValidFPValueForImm8(value),
"Value cannot be encoded into an 8-bit immediate");
const uint32_t bits = vixl::Float16ToRawbits(value);
const uint32_t sign = (bits & 0x8000) >> 8;
const uint32_t expb2 = (bits & 0x2000) >> 7;
const uint32_t b5_to_0 = (bits >> 6) & 0x3F;
return sign | expb2 | b5_to_0;
}
static uint32_t FP32ToImm8(float value) {
LOGMAN_THROW_A_FMT(IsValidFPValueForImm8(value),
"Value ({}) cannot be encoded into an 8-bit immediate", value);
@@ -33,7 +33,7 @@ public:
ASIMDScalarCopy(Op, 1, imm5, 0b0000, rd, rn);
}
void mov(FEXCore::ARMEmitter::ScalarRegSize size, FEXCore::ARMEmitter::VRegister rd, FEXCore::ARMEmitter::VRegister rn, uint32_t Index) {
void mov(ARMEmitter::ScalarRegSize size, ARMEmitter::VRegister rd, ARMEmitter::VRegister rn, uint32_t Index) {
dup(size, rd, rn, Index);
}
@@ -798,6 +798,52 @@ public:
// XXX:
//
// Floating-point data-processing (1 source)
void fmov(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b000000, rd, rn);
}
void fabs(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b000001, rd, rn);
}
void fneg(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b000010, rd, rn);
}
void fsqrt(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b000011, rd, rn);
}
void frintn(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b001000, rd, rn);
}
void frintp(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b001001, rd, rn);
}
void frintm(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b001010, rd, rn);
}
void frintz(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b001011, rd, rn);
}
void frinta(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b001100, rd, rn);
}
void frintx(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b001110, rd, rn);
}
void frinti(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b001111, rd, rn);
}
void frint32z(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b010000, rd, rn);
}
void frint32x(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b010001, rd, rn);
}
void frint64z(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b010010, rd, rn);
}
void frint64x(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b010011, rd, rn);
}
void fmov(SRegister rd, SRegister rn) {
Float1Source(0, 0, 0b00, 0b000000, rd.V(), rn.V());
}
@@ -1006,21 +1052,21 @@ public:
}
// Floating-point immediate
void fmov(FEXCore::ARMEmitter::ScalarRegSize size, FEXCore::ARMEmitter::VRegister rd, float Value) {
void fmov(ARMEmitter::ScalarRegSize size, ARMEmitter::VRegister rd, float Value) {
uint32_t M = 0;
uint32_t S = 0;
uint32_t ptype;
uint32_t imm8;
uint32_t imm5 = 0b0'0000;
if (size == FEXCore::ARMEmitter::ScalarRegSize::i16Bit) {
ptype = 0b11;
imm8 = FP16ToImm8(vixl::Float16(Value));
if (size == ARMEmitter::ScalarRegSize::i16Bit) {
LOGMAN_MSG_A_FMT("Unsupported");
FEX_UNREACHABLE;
}
else if (size == FEXCore::ARMEmitter::ScalarRegSize::i32Bit) {
else if (size == ARMEmitter::ScalarRegSize::i32Bit) {
ptype = 0b00;
imm8 = FP32ToImm8(Value);
}
else if (size == FEXCore::ARMEmitter::ScalarRegSize::i64Bit) {
else if (size == ARMEmitter::ScalarRegSize::i64Bit) {
ptype = 0b01;
imm8 = FP64ToImm8(Value);
}
@@ -1031,7 +1077,7 @@ public:
FloatScalarImmediate(M, S, ptype, imm8, imm5, rd);
}
void FloatScalarImmediate(uint32_t M, uint32_t S, uint32_t ptype, uint32_t imm8, uint32_t imm5, FEXCore::ARMEmitter::VRegister rd) {
void FloatScalarImmediate(uint32_t M, uint32_t S, uint32_t ptype, uint32_t imm8, uint32_t imm5, ARMEmitter::VRegister rd) {
constexpr uint32_t Op = 0b0001'1110'0010'0000'0001'00 << 10;
uint32_t Instr = Op;
@@ -1065,6 +1111,34 @@ public:
}
// Floating-point data-processing (2 source)
void fmul(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0000, rd, rn, rm);
}
void fdiv(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0001, rd, rn, rm);
}
void fadd(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0010, rd, rn, rm);
}
void fsub(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0011, rd, rn, rm);
}
void fmax(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0100, rd, rn, rm);
}
void fmin(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0101, rd, rn, rm);
}
void fmaxnm(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0110, rd, rn, rm);
}
void fminnm(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0111, rd, rn, rm);
}
void fnmul(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b1000, rd, rn, rm);
}
void fmul(SRegister rd, SRegister rn, SRegister rm) {
Float2Source(0, 0, 0b00, 0b0000, rd.V(), rn.V(), rm.V());
}
@@ -1150,6 +1224,16 @@ public:
}
// Floating-point conditional select
void fcsel(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm, Condition Cond) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for {}", __func__);
const uint32_t ConvertedSize =
size == ScalarRegSize::i64Bit ? 0b01 :
size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
FloatConditionalSelect(0, 0, ConvertedSize, rd, rn, rm, Cond);
}
void fcsel(SRegister rd, SRegister rn, SRegister rm, Condition Cond) {
FloatConditionalSelect(0, 0, 0b00, rd.V(), rn.V(), rm.V(), Cond);
}
@@ -1202,7 +1286,7 @@ public:
private:
// Advanced SIMD scalar copy
void ASIMDScalarCopy(uint32_t Op, uint32_t Q, uint32_t imm5, uint32_t imm4, FEXCore::ARMEmitter::VRegister rd, FEXCore::ARMEmitter::VRegister rn) {
void ASIMDScalarCopy(uint32_t Op, uint32_t Q, uint32_t imm5, uint32_t imm4, ARMEmitter::VRegister rd, ARMEmitter::VRegister rn) {
uint32_t Instr = Op;
Instr |= Q << 30;
@@ -1305,6 +1389,16 @@ private:
dc32(Instr);
}
void Float1Source(ScalarRegSize size, uint32_t M, uint32_t S, uint32_t opcode, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for {}", __func__);
const uint32_t ConvertedSize =
size == ScalarRegSize::i64Bit ? 0b01 :
size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
Float1Source(M, S, ConvertedSize, opcode, rd, rn);
}
// Floating-point compare
void FloatCompare(uint32_t M, uint32_t S, uint32_t ftype, uint32_t op, uint32_t opcode2, VRegister rn, VRegister rm) {
uint32_t Instr = 0b0001'1110'0010'0000'0010'0000'0000'0000;
@@ -1337,6 +1431,7 @@ private:
dc32(Instr);
}
// Floating-point data-processing (2 source)
void Float2Source(uint32_t M, uint32_t S, uint32_t ptype, uint32_t opcode, VRegister rd, VRegister rn, VRegister rm) {
uint32_t Instr = 0b0001'1110'0010'0000'0000'1000'0000'0000;
@@ -1351,6 +1446,16 @@ private:
dc32(Instr);
}
void Float2Source(ScalarRegSize size, uint32_t M, uint32_t S, uint32_t opcode, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for {}", __func__);
const uint32_t ConvertedSize =
size == ScalarRegSize::i64Bit ? 0b01 :
size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
Float2Source(M, S, ConvertedSize, opcode, rd, rn, rm);
}
// Floating-point conditional select
void FloatConditionalSelect(uint32_t M, uint32_t S, uint32_t ptype, VRegister rd, VRegister rn, VRegister rm, Condition Cond) {
uint32_t Instr = 0b0001'1110'0010'0000'0000'1100'0000'0000;
@@ -11,7 +11,7 @@ public:
// TODO: AT
// TODO: CFP
// TODO: CPP
void dc(FEXCore::ARMEmitter::DataCacheOperation DCOp, FEXCore::ARMEmitter::Register rt) {
void dc(ARMEmitter::DataCacheOperation DCOp, ARMEmitter::Register rt) {
constexpr uint32_t Op = 0b1101'0101'0000'1000'0111 << 12;
SystemInstruction(Op, 0, FEXCore::ToUnderlying(DCOp), rt);
}
@@ -48,67 +48,67 @@ public:
ExceptionGeneration(0b101, 0b000, 0b11, Imm);
}
// System instructions with register argument
void wfet(FEXCore::ARMEmitter::Register rt) {
void wfet(ARMEmitter::Register rt) {
SystemInstructionWithReg(0b0000, 0b000, rt);
}
void wfit(FEXCore::ARMEmitter::Register rt) {
void wfit(ARMEmitter::Register rt) {
SystemInstructionWithReg(0b0000, 0b001, rt);
}
// Hints
void nop() {
Hint(FEXCore::ARMEmitter::HintRegister::NOP);
Hint(ARMEmitter::HintRegister::NOP);
}
void yield() {
Hint(FEXCore::ARMEmitter::HintRegister::YIELD);
Hint(ARMEmitter::HintRegister::YIELD);
}
void wfe() {
Hint(FEXCore::ARMEmitter::HintRegister::WFE);
Hint(ARMEmitter::HintRegister::WFE);
}
void wfi() {
Hint(FEXCore::ARMEmitter::HintRegister::WFI);
Hint(ARMEmitter::HintRegister::WFI);
}
void sev() {
Hint(FEXCore::ARMEmitter::HintRegister::SEV);
Hint(ARMEmitter::HintRegister::SEV);
}
void sevl() {
Hint(FEXCore::ARMEmitter::HintRegister::SEVL);
Hint(ARMEmitter::HintRegister::SEVL);
}
void dgh() {
Hint(FEXCore::ARMEmitter::HintRegister::DGH);
Hint(ARMEmitter::HintRegister::DGH);
}
void csdb() {
Hint(FEXCore::ARMEmitter::HintRegister::CSDB);
Hint(ARMEmitter::HintRegister::CSDB);
}
// Barriers
void clrex(uint32_t imm = 15) {
LOGMAN_THROW_AA_FMT(imm < 16, "Immediate out of range");
Barrier(FEXCore::ARMEmitter::BarrierRegister::CLREX, imm);
Barrier(ARMEmitter::BarrierRegister::CLREX, imm);
}
void dsb(FEXCore::ARMEmitter::BarrierScope Scope) {
Barrier(FEXCore::ARMEmitter::BarrierRegister::DSB, FEXCore::ToUnderlying(Scope));
void dsb(ARMEmitter::BarrierScope Scope) {
Barrier(ARMEmitter::BarrierRegister::DSB, FEXCore::ToUnderlying(Scope));
}
void dmb(FEXCore::ARMEmitter::BarrierScope Scope) {
Barrier(FEXCore::ARMEmitter::BarrierRegister::DMB, FEXCore::ToUnderlying(Scope));
void dmb(ARMEmitter::BarrierScope Scope) {
Barrier(ARMEmitter::BarrierRegister::DMB, FEXCore::ToUnderlying(Scope));
}
void isb() {
Barrier(FEXCore::ARMEmitter::BarrierRegister::ISB, FEXCore::ToUnderlying(FEXCore::ARMEmitter::BarrierScope::SY));
Barrier(ARMEmitter::BarrierRegister::ISB, FEXCore::ToUnderlying(ARMEmitter::BarrierScope::SY));
}
void sb() {
Barrier(FEXCore::ARMEmitter::BarrierRegister::SB, 0);
Barrier(ARMEmitter::BarrierRegister::SB, 0);
}
void tcommit() {
Barrier(FEXCore::ARMEmitter::BarrierRegister::TCOMMIT, 0);
Barrier(ARMEmitter::BarrierRegister::TCOMMIT, 0);
}
// System register move
void msr(FEXCore::ARMEmitter::SystemRegister reg, FEXCore::ARMEmitter::Register rt) {
void msr(ARMEmitter::SystemRegister reg, ARMEmitter::Register rt) {
constexpr uint32_t Op = 0b1101'0101'0001 << 20;
SystemRegisterMove(Op, rt, reg);
}
void mrs(FEXCore::ARMEmitter::Register rd, FEXCore::ARMEmitter::SystemRegister reg) {
void mrs(ARMEmitter::Register rd, ARMEmitter::SystemRegister reg) {
constexpr uint32_t Op = 0b1101'0101'0011 << 20;
SystemRegisterMove(Op, rd, reg);
}
@@ -130,7 +130,7 @@ private:
}
// System instructions with register argument
void SystemInstructionWithReg(uint32_t CRm, uint32_t op2, FEXCore::ARMEmitter::Register rt) {
void SystemInstructionWithReg(uint32_t CRm, uint32_t op2, ARMEmitter::Register rt) {
uint32_t Instr = 0b1101'0101'0000'0011'0001 << 12;
Instr |= CRm << 8;
@@ -140,13 +140,13 @@ private:
}
// Hints
void Hint(FEXCore::ARMEmitter::HintRegister Reg) {
void Hint(ARMEmitter::HintRegister Reg) {
uint32_t Instr = 0b1101'0101'0000'0011'0010'0000'0001'1111U;
Instr |= FEXCore::ToUnderlying(Reg);
dc32(Instr);
}
// Barriers
void Barrier(FEXCore::ARMEmitter::BarrierRegister Reg, uint32_t CRm) {
void Barrier(ARMEmitter::BarrierRegister Reg, uint32_t CRm) {
uint32_t Instr = 0b1101'0101'0000'0011'0011'0000'0001'1111U;
Instr |= CRm << 8;
Instr |= FEXCore::ToUnderlying(Reg);
@@ -154,7 +154,7 @@ private:
}
// System Instruction
void SystemInstruction(uint32_t Op, uint32_t L, uint32_t SubOp, FEXCore::ARMEmitter::Register rt) {
void SystemInstruction(uint32_t Op, uint32_t L, uint32_t SubOp, ARMEmitter::Register rt) {
uint32_t Instr = Op;
Instr |= L << 21;
@@ -165,7 +165,7 @@ private:
}
// System register move
void SystemRegisterMove(uint32_t Op, FEXCore::ARMEmitter::Register rt, FEXCore::ARMEmitter::SystemRegister reg) {
void SystemRegisterMove(uint32_t Op, ARMEmitter::Register rt, ARMEmitter::SystemRegister reg) {
uint32_t Instr = Op;
Instr |= FEXCore::ToUnderlying(reg);
+311
View File
@@ -0,0 +1,311 @@
// Collection of utilities from vixl.
// Following is the vixl license.
// Copyright 2015, VIXL authors
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
//
// * Redistributions of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
// * Neither the name of ARM Limited nor the names of its contributors may be
// used to endorse or promote products derived from this software without
// specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS CONTRIBUTORS "AS IS" AND
// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
// Test if a given value can be encoded in the immediate field of a logical
// instruction.
// If it can be encoded, the function returns true, and values pointed to by n,
// imm_s and imm_r are updated with immediates encoded in the format required
// by the corresponding fields in the logical instruction.
// If it can not be encoded, the function returns false, and the values pointed
// to by n, imm_s and imm_r are undefined.
static bool IsImmLogical(uint64_t value,
unsigned width,
unsigned* n,
unsigned* imm_s,
unsigned* imm_r) {
[[maybe_unused]] constexpr auto kBRegSize = 8;
[[maybe_unused]] constexpr auto kHRegSize = 16;
[[maybe_unused]] constexpr auto kSRegSize = 32;
[[maybe_unused]] constexpr auto kDRegSize = 64;
constexpr auto kWRegSize = 32;
constexpr auto kXRegSize = 64;
LOGMAN_THROW_A_FMT((width == kBRegSize) || (width == kHRegSize) ||
(width == kSRegSize) || (width == kDRegSize), "Unexpected imm size");
bool negate = false;
// Logical immediates are encoded using parameters n, imm_s and imm_r using
// the following table:
//
// N imms immr size S R
// 1 ssssss rrrrrr 64 UInt(ssssss) UInt(rrrrrr)
// 0 0sssss xrrrrr 32 UInt(sssss) UInt(rrrrr)
// 0 10ssss xxrrrr 16 UInt(ssss) UInt(rrrr)
// 0 110sss xxxrrr 8 UInt(sss) UInt(rrr)
// 0 1110ss xxxxrr 4 UInt(ss) UInt(rr)
// 0 11110s xxxxxr 2 UInt(s) UInt(r)
// (s bits must not be all set)
//
// A pattern is constructed of size bits, where the least significant S+1 bits
// are set. The pattern is rotated right by R, and repeated across a 32 or
// 64-bit value, depending on destination register width.
//
// Put another way: the basic format of a logical immediate is a single
// contiguous stretch of 1 bits, repeated across the whole word at intervals
// given by a power of 2. To identify them quickly, we first locate the
// lowest stretch of 1 bits, then the next 1 bit above that; that combination
// is different for every logical immediate, so it gives us all the
// information we need to identify the only logical immediate that our input
// could be, and then we simply check if that's the value we actually have.
//
// (The rotation parameter does give the possibility of the stretch of 1 bits
// going 'round the end' of the word. To deal with that, we observe that in
// any situation where that happens the bitwise NOT of the value is also a
// valid logical immediate. So we simply invert the input whenever its low bit
// is set, and then we know that the rotated case can't arise.)
if (value & 1) {
// If the low bit is 1, negate the value, and set a flag to remember that we
// did (so that we can adjust the return values appropriately).
negate = true;
value = ~value;
}
if (width <= kWRegSize) {
// To handle 8/16/32-bit logical immediates, the very easiest thing is to repeat
// the input value to fill a 64-bit word. The correct encoding of that as a
// logical immediate will also be the correct encoding of the value.
// Avoid making the assumption that the most-significant 56/48/32 bits are zero by
// shifting the value left and duplicating it.
for (unsigned bits = width; bits <= kWRegSize; bits *= 2) {
value <<= bits;
uint64_t mask = (UINT64_C(1) << bits) - 1;
value |= ((value >> bits) & mask);
}
}
// The basic analysis idea: imagine our input word looks like this.
//
// 0011111000111110001111100011111000111110001111100011111000111110
// c b a
// |<--d-->|
//
// We find the lowest set bit (as an actual power-of-2 value, not its index)
// and call it a. Then we add a to our original number, which wipes out the
// bottommost stretch of set bits and replaces it with a 1 carried into the
// next zero bit. Then we look for the new lowest set bit, which is in
// position b, and subtract it, so now our number is just like the original
// but with the lowest stretch of set bits completely gone. Now we find the
// lowest set bit again, which is position c in the diagram above. Then we'll
// measure the distance d between bit positions a and c (using CLZ), and that
// tells us that the only valid logical immediate that could possibly be equal
// to this number is the one in which a stretch of bits running from a to just
// below b is replicated every d bits.
uint64_t a = LowestSetBit(value);
uint64_t value_plus_a = value + a;
uint64_t b = LowestSetBit(value_plus_a);
uint64_t value_plus_a_minus_b = value_plus_a - b;
uint64_t c = LowestSetBit(value_plus_a_minus_b);
int d, clz_a, out_n;
uint64_t mask;
if (c != 0) {
// The general case, in which there is more than one stretch of set bits.
// Compute the repeat distance d, and set up a bitmask covering the basic
// unit of repetition (i.e. a word with the bottom d bits set). Also, in all
// of these cases the N bit of the output will be zero.
clz_a = CountLeadingZeros(a, kXRegSize);
int clz_c = CountLeadingZeros(c, kXRegSize);
d = clz_a - clz_c;
mask = ((UINT64_C(1) << d) - 1);
out_n = 0;
} else {
// Handle degenerate cases.
//
// If any of those 'find lowest set bit' operations didn't find a set bit at
// all, then the word will have been zero thereafter, so in particular the
// last lowest_set_bit operation will have returned zero. So we can test for
// all the special case conditions in one go by seeing if c is zero.
if (a == 0) {
// The input was zero (or all 1 bits, which will come to here too after we
// inverted it at the start of the function), for which we just return
// false.
return false;
} else {
// Otherwise, if c was zero but a was not, then there's just one stretch
// of set bits in our word, meaning that we have the trivial case of
// d == 64 and only one 'repetition'. Set up all the same variables as in
// the general case above, and set the N bit in the output.
clz_a = CountLeadingZeros(a, kXRegSize);
d = 64;
mask = ~UINT64_C(0);
out_n = 1;
}
}
// If the repeat period d is not a power of two, it can't be encoded.
if (!IsPowerOf2(d)) {
return false;
}
if (((b - a) & ~mask) != 0) {
// If the bit stretch (b - a) does not fit within the mask derived from the
// repeat period, then fail.
return false;
}
// The only possible option is b - a repeated every d bits. Now we're going to
// actually construct the valid logical immediate derived from that
// specification, and see if it equals our original input.
//
// To repeat a value every d bits, we multiply it by a number of the form
// (1 + 2^d + 2^(2d) + ...), i.e. 0x0001000100010001 or similar. These can
// be derived using a table lookup on CLZ(d).
static const uint64_t multipliers[] = {
0x0000000000000001UL,
0x0000000100000001UL,
0x0001000100010001UL,
0x0101010101010101UL,
0x1111111111111111UL,
0x5555555555555555UL,
};
uint64_t multiplier = multipliers[CountLeadingZeros(d, kXRegSize) - 57];
uint64_t candidate = (b - a) * multiplier;
if (value != candidate) {
// The candidate pattern doesn't match our input value, so fail.
return false;
}
// We have a match! This is a valid logical immediate, so now we have to
// construct the bits and pieces of the instruction encoding that generates
// it.
// Count the set bits in our basic stretch. The special case of clz(0) == -1
// makes the answer come out right for stretches that reach the very top of
// the word (e.g. numbers like 0xffffc00000000000).
int clz_b = (b == 0) ? -1 : CountLeadingZeros(b, kXRegSize);
int s = clz_a - clz_b;
// Decide how many bits to rotate right by, to put the low bit of that basic
// stretch in position a.
int r;
if (negate) {
// If we inverted the input right at the start of this function, here's
// where we compensate: the number of set bits becomes the number of clear
// bits, and the rotation count is based on position b rather than position
// a (since b is the location of the 'lowest' 1 bit after inversion).
s = d - s;
r = (clz_b + 1) & (d - 1);
} else {
r = (clz_a + 1) & (d - 1);
}
// Now we're done, except for having to encode the S output in such a way that
// it gives both the number of set bits and the length of the repeated
// segment. The s field is encoded like this:
//
// imms size S
// ssssss 64 UInt(ssssss)
// 0sssss 32 UInt(sssss)
// 10ssss 16 UInt(ssss)
// 110sss 8 UInt(sss)
// 1110ss 4 UInt(ss)
// 11110s 2 UInt(s)
//
// So we 'or' (2 * -d) with our computed s to form imms.
if ((n != NULL) || (imm_s != NULL) || (imm_r != NULL)) {
*n = out_n;
*imm_s = ((2 * -d) | (s - 1)) & 0x3f;
*imm_r = r;
}
return true;
}
private:
template <typename V>
static inline bool IsPowerOf2(V value) {
return (value != 0) && ((value & (value - 1)) == 0);
}
// Some compilers dislike negating unsigned integers,
// so we provide an equivalent.
template <typename T>
static inline T UnsignedNegate(T value) {
static_assert(std::is_unsigned<T>::value);
return ~value + 1;
}
static inline uint64_t LowestSetBit(uint64_t value) {
return value & UnsignedNegate(value);
}
template <typename V>
static inline int CountLeadingZeros(V value, int width = (sizeof(V) * 8)) {
#if COMPILER_HAS_BUILTIN_CLZ
if (width == 32) {
return (value == 0) ? 32 : __builtin_clz(static_cast<unsigned>(value));
} else if (width == 64) {
return (value == 0) ? 64 : __builtin_clzll(value);
}
#endif
return CountLeadingZerosFallBack(value, width);
}
static inline int CountLeadingZerosFallBack(uint64_t value, int width) {
LOGMAN_THROW_A_FMT(IsPowerOf2(width) && (width <= 64), "Invalid width");
if (value == 0) {
return width;
}
int count = 0;
value = value << (64 - width);
if ((value & UINT64_C(0xffffffff00000000)) == 0) {
count += 32;
value = value << 32;
}
if ((value & UINT64_C(0xffff000000000000)) == 0) {
count += 16;
value = value << 16;
}
if ((value & UINT64_C(0xff00000000000000)) == 0) {
count += 8;
value = value << 8;
}
if ((value & UINT64_C(0xf000000000000000)) == 0) {
count += 4;
value = value << 4;
}
if ((value & UINT64_C(0xc000000000000000)) == 0) {
count += 2;
value = value << 2;
}
if ((value & UINT64_C(0x8000000000000000)) == 0) {
count += 1;
}
count += (value == 0);
return count;
}
public:
+3 -2
View File
@@ -1,5 +1,6 @@
{
"Config": {
"AdditionalArguments": "--no-sandbox"
"Comment": "Bypasses libGL's glX and instead sends GLX requests directly via xcb",
"ThunksDB": {
"GL": 0
}
}
-9
View File
@@ -2,9 +2,6 @@
"DB": {
"GL": {
"Library" : "libGL-guest.so",
"Depends": [
"X11"
],
"Overlay": [
"@PREFIX_LIB@/libGL.so",
"@PREFIX_LIB@/libGL.so.1",
@@ -33,16 +30,10 @@
},
"Vulkan": {
"Library": "libvulkan-guest.so",
"Depends": [
"xcb"
],
"Overlay": [
"@PREFIX_LIB@/libvulkan.so",
"@PREFIX_LIB@/libvulkan.so.1",
"@HOME@/.local/share/Steam/ubuntu12_32/steam-runtime/pinned_libs_64/libvulkan.so.1"
],
"Comment": [
"Vulkan library relies on xcb, otherwise it crashes with jemalloc"
]
},
"xcb": {
+5 -4
View File
@@ -3,11 +3,12 @@ FROM ubuntu:20.04 as builder
RUN DEBIAN_FRONTEND="noninteractive" apt-get update
RUN DEBIAN_FRONTEND="noninteractive" apt install -y cmake \
clang-10 llvm-10 nasm ninja-build \
libcap-dev libglfw3-dev libepoxy-dev python3-dev \
python3 linux-headers-generic
clang-10 llvm-10 nasm ninja-build pkg-config \
libcap-dev libglfw3-dev libepoxy-dev python3-dev libsdl2-dev \
python3 linux-headers-generic \
git
COPY . /opt/FEX
RUN git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
CMD [ "mkdir /opt/FEX/build" ]
+1 -1
-13
View File
@@ -1,13 +0,0 @@
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 2, December 2004
Copyright (C) 2018 Ryan Houdek <Sonicadvance1@gmail.com>
Everyone is permitted to copy and distribute verbatim or modified
copies of this license document, and changing it is allowed as long
as the name is changed.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. You just DO WHAT THE FUCK YOU WANT TO.
+336
View File
@@ -0,0 +1,336 @@
#!/usr/bin/env python3
#
# ====- code-format-helper, runs code formatters from the ci or in a hook --*- python -*--==#
#
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
# ==--------------------------------------------------------------------------------------==#
import argparse
import os
import subprocess
import sys
from typing import List, Optional
"""
This script is run by GitHub actions to ensure that the code in PR's conform to
the coding style of LLVM. It can also be installed as a pre-commit git hook to
check the coding style before submitting it. The canonical source of this script
is in the LLVM source tree under llvm/utils/git.
For C/C++ code it uses clang-format.
You can learn more about the LLVM coding style on llvm.org:
https://llvm.org/docs/CodingStandards.html
You can install this script as a git hook by symlinking it to the .git/hooks
directory:
ln -s $(pwd)/llvm/utils/git/code-format-helper.py .git/hooks/pre-commit
You can control the exact path to clang-format with the following
environment variable: $CLANG_FORMAT_PATH.
"""
class FormatArgs:
start_rev: str = None
end_rev: str = None
repo: str = None
changed_files: List[str] = []
token: str = None
verbose: bool = True
issue_number: int = 0
write_comment_to_file: str = None
def __init__(self, args: argparse.Namespace = None) -> None:
if not args is None:
self.start_rev = args.start_rev
self.end_rev = args.end_rev
self.repo = args.repo
self.token = args.token
self.changed_files = args.changed_files
self.issue_number = args.issue_number
self.write_comment_to_file = args.write_comment_to_file
class FormatHelper:
COMMENT_TAG = "<!--CODE FORMAT COMMENT: {fmt}-->"
name: str
friendly_name: str
comment: dict = None
@property
def comment_tag(self) -> str:
return self.COMMENT_TAG.replace("fmt", self.name)
@property
def instructions(self) -> str:
raise NotImplementedError()
def has_tool(self) -> bool:
raise NotImplementedError()
def format_run(self, changed_files: List[str], args: FormatArgs) -> Optional[str]:
raise NotImplementedError()
def pr_comment_text_for_diff(self, diff: str) -> str:
return f"""
:warning: {self.friendly_name}, {self.name} found issues in your code. :warning:
<details>
<summary>
You can test this locally with the following command:
</summary>
``````````bash
{self.instructions}
``````````
</details>
<details>
<summary>
View the diff from {self.name} here.
</summary>
``````````diff
{diff}
``````````
</details>
"""
# TODO: any type should be replaced with the correct github type, but it requires refactoring to
# not require the github module to be installed everywhere.
def find_comment(self, pr: any) -> any:
for comment in pr.as_issue().get_comments():
if self.comment_tag in comment.body:
return comment
return None
def update_pr(self, comment_text: str, args: FormatArgs, create_new: bool) -> None:
import github
from github import IssueComment, PullRequest
repo = github.Github(args.token).get_repo(args.repo)
pr = repo.get_issue(args.issue_number).as_pull_request()
comment_text = self.comment_tag + "\n\n" + comment_text
existing_comment = self.find_comment(pr)
if args.write_comment_to_file:
if create_new or existing_comment:
self.comment = {"body": comment_text}
if existing_comment:
self.comment["id"] = existing_comment.id
return
if existing_comment:
existing_comment.edit(comment_text)
elif create_new:
pr.as_issue().create_comment(comment_text)
def run(self, changed_files: List[str], args: FormatArgs) -> bool:
changed_files = [arg for arg in changed_files if "third-party" not in arg]
diff = self.format_run(changed_files, args)
should_update_gh = args.token is not None and args.repo is not None
if diff is None:
if should_update_gh:
comment_text = (
":white_check_mark: With the latest revision "
f"this PR passed the {self.friendly_name}."
)
self.update_pr(comment_text, args, create_new=False)
return True
elif len(diff) > 0:
if should_update_gh:
comment_text = self.pr_comment_text_for_diff(diff)
self.update_pr(comment_text, args, create_new=True)
else:
print(
f"Warning: {self.friendly_name}, {self.name} detected "
"some issues with your code formatting..."
)
return False
else:
# The formatter failed but didn't output a diff (e.g. some sort of
# infrastructure failure).
comment_text = (
f":warning: The {self.friendly_name} failed without printing "
"a diff. Check the logs for stderr output. :warning:"
)
self.update_pr(comment_text, args, create_new=False)
return False
class ClangFormatHelper(FormatHelper):
name = "clang-format"
friendly_name = "C/C++ code formatter"
@property
def cformat_wrapper_path(self) -> str:
relpath = "../../Scripts/clang-format.py"
curpath = os.path.dirname(os.path.abspath(__file__))
return os.path.abspath(os.path.normpath(os.path.join(curpath, relpath)))
@property
def instructions(self) -> str:
return " ".join(self.cf_cmd)
def should_include_extensionless_file(self, path: str) -> bool:
return path.startswith("libcxx/include")
def filter_changed_files(self, changed_files: List[str]) -> List[str]:
filtered_files = []
for path in changed_files:
_, ext = os.path.splitext(path)
if ext in (".cpp", ".c", ".h", ".hpp", ".hxx", ".cxx", ".inc", ".cppm"):
filtered_files.append(path)
elif ext == "" and self.should_include_extensionless_file(path):
filtered_files.append(path)
return filtered_files
@property
def clang_fmt_path(self) -> str:
if "CLANG_FORMAT_PATH" in os.environ:
return os.environ["CLANG_FORMAT_PATH"]
return "git-clang-format"
def has_tool(self) -> bool:
cmd = [self.clang_fmt_path, "-h"]
proc = None
try:
proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except:
return False
return proc.returncode == 0
def format_run(self, changed_files: List[str], args: FormatArgs) -> Optional[str]:
cpp_files = self.filter_changed_files(changed_files)
if not cpp_files:
return None
cf_cmd = [
self.clang_fmt_path,
f"--binary={self.cformat_wrapper_path}",
"--diff",
]
if args.start_rev and args.end_rev:
cf_cmd.append(args.start_rev)
cf_cmd.append(args.end_rev)
cf_cmd.append("--")
cf_cmd += cpp_files
if args.verbose:
print(f"Running: {' '.join(cf_cmd)}")
self.cf_cmd = cf_cmd
proc = subprocess.run(cf_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
sys.stdout.write(proc.stderr.decode("utf-8"))
if proc.returncode != 0:
# formatting needed, or the command otherwise failed
if args.verbose:
print(f"error: {self.name} exited with code {proc.returncode}")
# Print the diff in the log so that it is viewable there
print(proc.stdout.decode("utf-8"))
return proc.stdout.decode("utf-8")
else:
return None
ALL_FORMATTERS = [ClangFormatHelper()]
def hook_main():
# fill out args
args = FormatArgs()
args.verbose = False
# find the changed files
cmd = ["git", "diff", "--cached", "--name-only", "--diff-filter=d"]
proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = proc.stdout.decode("utf-8")
for line in output.splitlines():
args.changed_files.append(line)
failed_fmts = []
for fmt in ALL_FORMATTERS:
if fmt.has_tool():
if not fmt.run(args.changed_files, args):
failed_fmts.append(fmt.name)
if fmt.comment:
comments.append(fmt.comment)
else:
print(f"Couldn't find {fmt.name}, can't check " + fmt.friendly_name.lower())
if len(failed_fmts) > 0:
sys.exit(1)
sys.exit(0)
if __name__ == "__main__":
script_path = os.path.abspath(__file__)
if ".git/hooks" in script_path:
hook_main()
sys.exit(0)
parser = argparse.ArgumentParser()
parser.add_argument(
"--token", type=str, required=False, help="GitHub authentication token"
)
parser.add_argument(
"--repo",
type=str,
default=os.getenv("GITHUB_REPOSITORY", "llvm/llvm-project"),
help="The GitHub repository that we are working with in the form of <owner>/<repo> (e.g. llvm/llvm-project)",
)
parser.add_argument("--issue-number", type=int, required=True)
parser.add_argument(
"--start-rev",
type=str,
required=True,
help="Compute changes from this revision.",
)
parser.add_argument(
"--end-rev", type=str, required=True, help="Compute changes to this revision"
)
parser.add_argument(
"--changed-files",
type=str,
help="Comma separated list of files that has been changed",
)
parser.add_argument(
"--write-comment-to-file",
type=str,
help="Don't post comments on the PR, instead write the comments and metadata a file",
)
args = FormatArgs(parser.parse_args())
changed_files = []
if args.changed_files:
changed_files = args.changed_files.split(",")
failed_formatters = []
comments = []
for fmt in ALL_FORMATTERS:
if not fmt.run(changed_files, args):
failed_formatters.append(fmt.name)
if fmt.comment:
comments.append(fmt.comment)
if len(comments):
with open(args.write_comment_to_file, "w") as f:
import json
json.dump(comments, f)
if len(failed_formatters) > 0:
print(f"error: some formatters failed: {' '.join(failed_formatters)}")
sys.exit(1)
+52
View File
@@ -0,0 +1,52 @@
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --output-file=llvm/utils/git/requirements_formatting.txt llvm/utils/git/requirements_formatting.txt.in
#
black==23.9.1
# via
# -r llvm/utils/git/requirements_formatting.txt.in
# darker
certifi==2023.7.22
# via requests
cffi==1.15.1
# via
# cryptography
# pynacl
charset-normalizer==3.2.0
# via requests
click==8.1.7
# via black
cryptography==41.0.3
# via pyjwt
darker==1.7.2
# via -r llvm/utils/git/requirements_formatting.txt.in
deprecated==1.2.14
# via pygithub
idna==3.4
# via requests
mypy-extensions==1.0.0
# via black
packaging==23.1
# via black
pathspec==0.11.2
# via black
platformdirs==3.10.0
# via black
pycparser==2.21
# via cffi
pygithub==1.59.1
# via -r llvm/utils/git/requirements_formatting.txt.in
pyjwt[crypto]==2.8.0
# via pygithub
pynacl==1.5.0
# via pygithub
requests==2.31.0
# via pygithub
toml==0.10.2
# via darker
urllib3==2.0.4
# via requests
wrapt==1.15.0
# via deprecated
+1 -1
+1 -1
+1 -1
+1 -1
+9 -5
View File
@@ -13,8 +13,6 @@ if (CMAKE_SYSTEM_PROCESSOR MATCHES "^aarch64|^arm64|^armv8\.*")
set(_M_ARM_64 1)
endif()
option(ENABLE_CLANG_FORMAT "Run clang format over the source" FALSE)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
cmake_policy(SET CMP0083 NEW) # Follow new PIE policy
include(CheckPIESupported)
@@ -30,15 +28,21 @@ set(CMAKE_REQUIRED_FLAGS "-std=c++11 -Wattributes -Werror=attributes")
check_cxx_source_compiles(
"
__attribute__((preserve_all))
void Testy() {
int Testy(int a, int b, int c, int d, int e, int f) {
return a + b + c + d + e + f;
}
int main() {
return 0;
return Testy(0, 1, 2, 3, 4, 5);
}"
HAS_CLANG_PRESERVE_ALL)
unset(CMAKE_REQUIRED_FLAGS)
if (HAS_CLANG_PRESERVE_ALL)
message(STATUS "Has clang::preserve_all")
if (MINGW_BUILD)
message(STATUS "Ignoring broken clang::preserve_all support")
set(HAS_CLANG_PRESERVE_ALL FALSE)
else()
message(STATUS "Has clang::preserve_all")
endif()
endif ()
if (EXISTS ${CMAKE_CURRENT_DIR}/External/vixl/)
+21
View File
@@ -441,6 +441,24 @@ def print_parse_envloader_options(options):
output_argloader.write("}\n")
output_argloader.write("#endif\n")
def print_parse_jsonloader_options(options):
output_argloader.write("#ifdef JSONLOADER\n")
output_argloader.write("#undef JSONLOADER\n")
output_argloader.write("if (false) {}\n")
for op_group, group_vals in options.items():
for op_key, op_vals in group_vals.items():
value_type = op_vals["Type"]
if (value_type == "strenum"):
output_argloader.write("else if (KeyName == \"{0}\") {{\n".format(op_key))
output_argloader.write("Set(KeyOption, FEXCore::Config::EnumParser<FEXCore::Config::{}ConfigPair>(FEXCore::Config::{}_EnumPairs, Value_View));\n".format(op_key, op_key, op_key))
output_argloader.write("}\n")
output_argloader.write("else {{\n".format(op_key))
output_argloader.write("Set(KeyOption, ConfigString);\n")
output_argloader.write("}\n")
output_argloader.write("#endif\n")
def print_parse_enum_options(options):
output_argloader.write("#ifdef ENUMDEFINES\n")
output_argloader.write("#undef ENUMDEFINES\n")
@@ -556,6 +574,9 @@ print_parse_argloader_options(options);
# Generate environment loader code
print_parse_envloader_options(options);
# Generate json loader code
print_parse_jsonloader_options(options);
# Generate enum variable options
print_parse_enum_options(options);
+120 -22
View File
@@ -46,11 +46,16 @@ class OpDefinition:
NumElements: str
OpClass: str
HasSideEffects: bool
ImplicitFlagClobber: bool
RAOverride: int
SwitchGen: bool
ArgPrinter: bool
SSAArgNum: int
NonSSAArgNum: int
DynamicDispatch: bool
JITDispatch: bool
JITDispatchOverride: str
TiedSource: int
Arguments: list
EmitValidation: list
Desc: list
@@ -64,11 +69,16 @@ class OpDefinition:
self.OpClass = None
self.OpSize = 0
self.HasSideEffects = False
self.ImplicitFlagClobber = False
self.RAOverride = -1
self.SwitchGen = True
self.ArgPrinter = True
self.SSAArgNum = 0
self.NonSSAArgNum = 0
self.DynamicDispatch = False
self.JITDispatch = True
self.JITDispatchOverride = None
self.TiedSource = -1
self.Arguments = []
self.EmitValidation = []
self.Desc = []
@@ -213,6 +223,9 @@ def parse_ops(ops):
if "HasSideEffects" in op_val:
OpDef.HasSideEffects = bool(op_val["HasSideEffects"])
if "ImplicitFlagClobber" in op_val:
OpDef.ImplicitFlagClobber = bool(op_val["ImplicitFlagClobber"])
if "ArgPrinter" in op_val:
OpDef.ArgPrinter = bool(op_val["ArgPrinter"])
@@ -228,6 +241,18 @@ def parse_ops(ops):
if "Desc" in op_val:
OpDef.Desc = op_val["Desc"]
if "DynamicDispatch" in op_val:
OpDef.DynamicDispatch = bool(op_val["DynamicDispatch"])
if "JITDispatch" in op_val:
OpDef.JITDispatch = bool(op_val["JITDispatch"])
if "JITDispatchOverride" in op_val:
OpDef.JITDispatchOverride = op_val["JITDispatchOverride"]
if "TiedSource" in op_val:
OpDef.TiedSource = op_val["TiedSource"]
# Do some fixups of the data here
if len(OpDef.EmitValidation) != 0:
for i in range(len(OpDef.EmitValidation)):
@@ -352,12 +377,30 @@ def print_ir_sizes():
output_file.write("[[maybe_unused, nodiscard]] static size_t GetSize(IROps Op) { return IRSizes[Op]; }\n\n")
output_file.write("[[nodiscard, gnu::const, gnu::visibility(\"default\")]] std::string_view const& GetName(IROps Op);\n")
output_file.write("[[nodiscard, gnu::const, gnu::visibility(\"default\")]] uint8_t GetArgs(IROps Op);\n")
output_file.write("[[nodiscard, gnu::const, gnu::visibility(\"default\")]] uint8_t GetRAArgs(IROps Op);\n")
output_file.write("[[nodiscard, gnu::const, gnu::visibility(\"default\")]] FEXCore::IR::RegisterClassType GetRegClass(IROps Op);\n\n")
output_file.write("[[nodiscard, gnu::const, gnu::visibility(\"default\")]] bool HasSideEffects(IROps Op);\n")
output_file.write("[[nodiscard, gnu::const, gnu::visibility(\"default\")]] bool GetHasDest(IROps Op);\n")
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] std::string_view const& GetName(IROps Op);\n'
)
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] uint8_t GetArgs(IROps Op);\n'
)
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] uint8_t GetRAArgs(IROps Op);\n'
)
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] FEXCore::IR::RegisterClassType GetRegClass(IROps Op);\n\n'
)
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] bool HasSideEffects(IROps Op);\n'
)
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] bool ImplicitFlagClobber(IROps Op);\n'
)
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] bool GetHasDest(IROps Op);\n'
)
output_file.write(
'[[nodiscard, gnu::const, gnu::visibility("default")]] int8_t TiedSource(IROps Op);\n'
)
output_file.write("#undef IROP_SIZES\n")
output_file.write("#endif\n\n")
@@ -450,15 +493,27 @@ def print_ir_getraargs():
def print_ir_hassideeffects():
output_file.write("#ifdef IROP_HASSIDEEFFECTS_IMPL\n")
output_file.write("constexpr std::array<uint8_t, OP_LAST + 1> SideEffects = {\n")
for op in IROps:
output_file.write("\t{},\n".format(("true" if op.HasSideEffects else "false")))
for array, prop, T in [
("SideEffects", "HasSideEffects", "bool"),
("ImplicitFlagClobbers", "ImplicitFlagClobber", "bool"),
("TiedSources", "TiedSource", "int8_t"),
]:
output_file.write(
f"constexpr std::array<{'uint8_t' if T == 'bool' else T}, OP_LAST + 1> {array} = {{\n"
)
for op in IROps:
if T == "bool":
output_file.write(
"\t{},\n".format(("true" if getattr(op, prop) else "false"))
)
else:
output_file.write(f"\t{getattr(op, prop)},\n")
output_file.write("};\n\n")
output_file.write("};\n\n")
output_file.write("bool HasSideEffects(IROps Op) {\n")
output_file.write(" return SideEffects[Op];\n")
output_file.write("}\n")
output_file.write(f"{T} {prop}(IROps Op) {{\n")
output_file.write(f" return {array}[Op];\n")
output_file.write("}\n")
output_file.write("#undef IROP_HASSIDEEFFECTS_IMPL\n")
output_file.write("#endif\n\n")
@@ -497,14 +552,20 @@ def print_ir_arg_printer():
output_file.write("\t*out << \" \";\n")
SSAArgNum = 0
FirstArg = True
for i in range(0, len(op.Arguments)):
arg = op.Arguments[i]
LastArg = len(op.Arguments) - i - 1 == 0
# No point printing temporaries that we can't recover
if arg.Temporary:
# Temporary that we can't recover
output_file.write("\t*out << \"{}:Tmp:{}\";\n".format(arg.Type, arg.Name))
elif arg.IsSSA:
continue
if FirstArg:
FirstArg = False
else:
output_file.write('\t*out << ", ";\n')
if arg.IsSSA:
# SSA value
output_file.write("\tPrintArg(out, IR, Op->Header.Args[{}], RAData);\n".format(SSAArgNum))
SSAArgNum = SSAArgNum + 1
@@ -512,9 +573,6 @@ def print_ir_arg_printer():
# User defined op that is stored
output_file.write("\tPrintArg(out, IR, Op->{});\n".format(arg.Name))
if not LastArg:
output_file.write("\t*out << \", \";\n")
output_file.write("break;\n")
output_file.write("}\n")
@@ -627,6 +685,10 @@ def print_ir_allocator_helpers():
output_file.write(") {\n")
# Save NZCV if needed before clobbering NZCV
if op.ImplicitFlagClobber:
output_file.write("\t\tSaveNZCV(IROps::OP_{});".format(op.Name.upper()))
output_file.write("\t\tauto Op = AllocateOp<IROp_{}, IROps::OP_{}>();\n".format(op.Name, op.Name.upper()))
if op.SSAArgNum != 0:
@@ -675,7 +737,8 @@ def print_ir_allocator_helpers():
output_file.write("\t\t#if defined(ASSERTIONS_ENABLED) && ASSERTIONS_ENABLED\n")
for Validation in op.EmitValidation:
output_file.write("\tLOGMAN_THROW_A_FMT({}, \"\");\n".format(Validation))
Sanitized = Validation.replace("\"", "\\\"")
output_file.write("\tLOGMAN_THROW_A_FMT({}, \"{}\");\n".format(Validation, Sanitized))
output_file.write("\t\t#endif\n")
output_file.write("\t\treturn Op;\n")
@@ -730,10 +793,38 @@ def print_ir_parser_switch_helper():
output_file.write("#undef IROP_PARSER_SWITCH_HELPERS\n")
output_file.write("#endif\n")
if (len(sys.argv) < 3):
def print_ir_dispatcher_defs():
output_dispatch_file.write("#ifdef IROP_DISPATCH_DEFS\n")
for op in IROps:
if op.Name != "Last" and op.SwitchGen and op.JITDispatch and op.JITDispatchOverride == None:
output_dispatch_file.write("DEF_OP({});\n".format(op.Name))
output_dispatch_file.write("#undef IROP_DISPATCH_DEFS\n")
output_dispatch_file.write("#endif\n")
def print_ir_dispatcher_dispatch():
output_dispatch_file.write("#ifdef IROP_DISPATCH_DISPATCH\n")
for op in IROps:
if op.Name != "Last" and op.JITDispatch:
DispatchName = op.Name
if op.JITDispatchOverride != None:
DispatchName = op.JITDispatchOverride
if (op.DynamicDispatch):
output_dispatch_file.write("REGISTER_OP_RT({}, {});\n".format(op.Name.upper(), DispatchName))
else:
output_dispatch_file.write("REGISTER_OP({}, {});\n".format(op.Name.upper(), DispatchName))
output_dispatch_file.write("#undef IROP_DISPATCH_DISPATCH\n")
output_dispatch_file.write("#endif\n")
if (len(sys.argv) < 4):
ExitError()
output_filename = sys.argv[2]
output_dispatcher_filename = sys.argv[3]
json_file = open(sys.argv[1], "r")
json_text = json_file.read()
json_file.close()
@@ -763,3 +854,10 @@ print_ir_allocator_helpers()
print_ir_parser_switch_helper()
output_file.close()
output_dispatch_file = open(output_dispatcher_filename, "w")
print_ir_dispatcher_defs()
print_ir_dispatcher_dispatch()
output_dispatch_file.close()
+11 -23
View File
@@ -7,6 +7,7 @@ set (FEXCORE_BASE_SRCS
Utils/FileLoading.cpp
Utils/ForcedAssert.cpp
Utils/LogManager.cpp
Utils/SpinWaitLock.cpp
)
if (NOT MINGW_BUILD)
@@ -90,20 +91,18 @@ set (SRCS
Interface/Core/CPUBackend.cpp
Interface/Core/CPUID.cpp
Interface/Core/Frontend.cpp
Interface/Core/GdbServer.cpp
Interface/Core/HostFeatures.cpp
Interface/Core/ObjectCache/JobHandling.cpp
Interface/Core/ObjectCache/NamedRegionObjectHandler.cpp
Interface/Core/ObjectCache/ObjectCacheService.cpp
Interface/Core/OpcodeDispatcher/AVX_128.cpp
Interface/Core/OpcodeDispatcher/Crypto.cpp
Interface/Core/OpcodeDispatcher/Flags.cpp
Interface/Core/OpcodeDispatcher/Vector.cpp
Interface/Core/OpcodeDispatcher/X87.cpp
Interface/Core/OpcodeDispatcher/X87F64.cpp
Interface/Core/OpcodeDispatcher.cpp
Interface/Core/SignalDelegator.cpp
Interface/Core/X86Tables.cpp
Interface/Core/X86DebugInfo.cpp
Interface/Core/X86HelperGen.cpp
Interface/Core/ArchHelpers/Arm64Emitter.cpp
Interface/Core/Dispatcher/Dispatcher.cpp
@@ -136,23 +135,16 @@ set (SRCS
Interface/GDBJIT/GDBJIT.cpp
Interface/IR/AOTIR.cpp
Interface/IR/IRDumper.cpp
Interface/IR/IRParser.cpp
Interface/IR/IREmitter.cpp
Interface/IR/PassManager.cpp
Interface/IR/Passes/ConstProp.cpp
Interface/IR/Passes/DeadCodeElimination.cpp
Interface/IR/Passes/DeadContextStoreElimination.cpp
Interface/IR/Passes/IRCompaction.cpp
Interface/IR/Passes/IRDumperPass.cpp
Interface/IR/Passes/IRValidation.cpp
Interface/IR/Passes/RAValidation.cpp
Interface/IR/Passes/LongDivideRemovalPass.cpp
Interface/IR/Passes/ValueDominanceValidation.cpp
Interface/IR/Passes/RedundantFlagCalculationElimination.cpp
Interface/IR/Passes/DeadStoreElimination.cpp
Interface/IR/Passes/RegisterAllocationPass.cpp
Interface/IR/Passes/InlineCallOptimization.cpp
Utils/NetStream.cpp
Utils/Telemetry.cpp
Utils/Threads.cpp
Utils/Profiler.cpp
@@ -197,12 +189,15 @@ endif()
# Some defines for the softfloat library
list(APPEND DEFINES "-DSOFTFLOAT_BUILTIN_CLZ")
set (LIBS fmt::fmt vixl xxhash FEXHeaderUtils)
set (LIBS fmt::fmt vixl xxHash::xxhash FEXHeaderUtils CodeEmitter)
if (NOT MINGW_BUILD)
list (APPEND LIBS dl)
else()
list (APPEND LIBS synchronization)
if (_M_ARM_64EC)
list (APPEND LIBS kernelbase)
endif()
endif()
if (ENABLE_JEMALLOC)
@@ -221,15 +216,16 @@ configure_file(
# Generate IR include file
set(OUTPUT_IR_FOLDER "${CMAKE_BINARY_DIR}/include/FEXCore/IR")
set(OUTPUT_NAME "${OUTPUT_IR_FOLDER}/IRDefines.inc")
set(OUTPUT_DISPATCHER_NAME "${OUTPUT_IR_FOLDER}/IRDefines_Dispatch.inc")
set(INPUT_NAME "${CMAKE_CURRENT_SOURCE_DIR}/Interface/IR/IR.json")
file(MAKE_DIRECTORY "${OUTPUT_IR_FOLDER}")
add_custom_command(
OUTPUT "${OUTPUT_NAME}"
OUTPUT "${OUTPUT_NAME}" "${OUTPUT_DISPATCHER_NAME}"
DEPENDS "${INPUT_NAME}"
DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/../Scripts/json_ir_generator.py"
COMMAND "python3" "${CMAKE_CURRENT_SOURCE_DIR}/../Scripts/json_ir_generator.py" "${INPUT_NAME}" "${OUTPUT_NAME}"
COMMAND "python3" "${CMAKE_CURRENT_SOURCE_DIR}/../Scripts/json_ir_generator.py" "${INPUT_NAME}" "${OUTPUT_NAME}" "${OUTPUT_DISPATCHER_NAME}"
)
set_source_files_properties(${OUTPUT_NAME} PROPERTIES
@@ -361,6 +357,7 @@ function(AddObject Name Type)
add_library(${Name} ${Type} ${SRCS})
target_link_libraries(${Name} FEXCore_Base)
target_compile_options(${Name} PRIVATE ${FEX_TUNE_COMPILE_FLAGS})
AddDefaultOptionsToTarget(${Name})
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
@@ -369,17 +366,8 @@ endfunction()
function(AddLibrary Name Type)
add_library(${Name} ${Type} $<TARGET_OBJECTS:${PROJECT_NAME}_object>)
target_link_libraries(${Name} FEXCore_Base)
target_compile_options(${Name} PRIVATE ${FEX_TUNE_COMPILE_FLAGS})
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
if (MINGW_BUILD)
# Mingw build isn't building a linux shared library, so it can't have a SONAME.
set_target_properties(${Name} PROPERTIES NO_SONAME ON)
# Change the suffixes otherwise cmake continues using .a and .so
if (${Type} STREQUAL SHARED)
set_target_properties(${Name} PROPERTIES SUFFIX ".dll")
elseif(${Type} STREQUAL STATIC)
set_target_properties(${Name} PROPERTIES SUFFIX ".lib")
endif()
endif()
AddDefaultOptionsToTarget(${Name})
endfunction()
+11 -9
View File
@@ -18,14 +18,14 @@ struct BitSet final {
constexpr static size_t MinimumSize = sizeof(ElementType);
constexpr static size_t MinimumSizeBits = sizeof(ElementType) * 8;
ElementType *Memory;
ElementType* Memory;
void Allocate(size_t Elements) {
size_t AllocateSize = AlignUp(Elements, MinimumSizeBits) / MinimumSize;
size_t AllocateSize = ToBytes(Elements);
LOGMAN_THROW_AA_FMT((AllocateSize * MinimumSize) >= Elements, "Fail");
Memory = static_cast<ElementType*>(FEXCore::Allocator::malloc(AllocateSize));
}
void Realloc(size_t Elements) {
size_t AllocateSize = AlignUp(Elements, MinimumSizeBits) / MinimumSize;
size_t AllocateSize = ToBytes(Elements);
LOGMAN_THROW_AA_FMT((AllocateSize * MinimumSize) >= Elements, "Fail");
Memory = static_cast<ElementType*>(FEXCore::Allocator::realloc(Memory, AllocateSize));
}
@@ -43,10 +43,13 @@ struct BitSet final {
Memory[Element / MinimumSizeBits] &= (1ULL << (Element % MinimumSizeBits));
}
void MemClear(size_t Elements) {
memset(Memory, 0, AlignUp(Elements / MinimumSizeBits, MinimumSizeBits));
memset(Memory, 0, ToBytes(Elements));
}
void MemSet(size_t Elements) {
memset(Memory, 0xFF, AlignUp(Elements / MinimumSizeBits, MinimumSizeBits));
memset(Memory, 0xFF, ToBytes(Elements));
}
uint32_t ToBytes(size_t Elements) {
return AlignUp(Elements, MinimumSizeBits) / MinimumSize;
}
// This very explicitly doesn't let you take an address
@@ -62,11 +65,10 @@ struct BitSetView final {
constexpr static size_t MinimumSize = sizeof(ElementType);
constexpr static size_t MinimumSizeBits = sizeof(ElementType) * 8;
ElementType *Memory;
ElementType* Memory;
void GetView(BitSet<T> &Set, uint64_t ElementOffset) {
LOGMAN_THROW_AA_FMT((ElementOffset % MinimumSize) == 0,
"Bitset view offset needs to be aligned to size of backing element");
void GetView(BitSet<T>& Set, uint64_t ElementOffset) {
LOGMAN_THROW_AA_FMT((ElementOffset % MinimumSize) == 0, "Bitset view offset needs to be aligned to size of backing element");
Memory = &Set.Memory[ElementOffset / MinimumSizeBits];
}
+116 -107
View File
@@ -7,133 +7,142 @@
#include <unistd.h>
namespace FEXCore {
JITSymbols::JITSymbols() {
}
JITSymbols::JITSymbols() {}
JITSymbols::~JITSymbols() {
if (fd != -1) {
close(fd);
}
JITSymbols::~JITSymbols() {
if (fd != -1) {
close(fd);
}
}
void JITSymbols::InitFile() {
// We can't use FILE here since we must be robust against forking processes closing our FD from under us.
void JITSymbols::InitFile() {
// We can't use FILE here since we must be robust against forking processes closing our FD from under us.
#ifdef __ANDROID__
// Android simpleperf looks in /data/local/tmp instead of /tmp
const auto PerfMap = fextl::fmt::format("/data/local/tmp/perf-{}.map", getpid());
// Android simpleperf looks in /data/local/tmp instead of /tmp
const auto PerfMap = fextl::fmt::format("/data/local/tmp/perf-{}.map", getpid());
#else
const auto PerfMap = fextl::fmt::format("/tmp/perf-{}.map", getpid());
const auto PerfMap = fextl::fmt::format("/tmp/perf-{}.map", getpid());
#endif
fd = open(PerfMap.c_str(), O_CREAT | O_TRUNC | O_WRONLY | O_APPEND, 0644);
fd = open(PerfMap.c_str(), O_CREAT | O_TRUNC | O_WRONLY | O_APPEND, 0644);
}
void JITSymbols::RegisterNamedRegion(const void* HostAddr, uint32_t CodeSize, std::string_view Name) {
if (fd == -1) {
return;
}
void JITSymbols::RegisterNamedRegion(const void *HostAddr, uint32_t CodeSize, std::string_view Name) {
if (fd == -1) return;
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto Buffer = fextl::fmt::format("{} {:x} {}\n", HostAddr, CodeSize, Name);
auto Result = write(fd, Buffer.c_str(), Buffer.size());
if (Result == -1 && errno == EBADF) {
fd = -1;
}
}
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto Buffer = fextl::fmt::format("{} {:x} {}\n", HostAddr, CodeSize, Name);
auto Result = write(fd, Buffer.c_str(), Buffer.size());
if (Result == -1 && errno == EBADF) {
fd = -1;
}
void JITSymbols::RegisterJITSpace(const void* HostAddr, uint32_t CodeSize) {
if (fd == -1) {
return;
}
void JITSymbols::RegisterJITSpace(const void *HostAddr, uint32_t CodeSize) {
if (fd == -1) return;
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto Buffer = fextl::fmt::format("{} {:x} FEXJIT\n", HostAddr, CodeSize);
auto Result = write(fd, Buffer.c_str(), Buffer.size());
if (Result == -1 && errno == EBADF) {
fd = -1;
}
}
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto Buffer = fextl::fmt::format("{} {:x} FEXJIT\n", HostAddr, CodeSize);
auto Result = write(fd, Buffer.c_str(), Buffer.size());
if (Result == -1 && errno == EBADF) {
fd = -1;
}
// Buffered JIT symbols.
void JITSymbols::Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint64_t GuestAddr, uint32_t CodeSize) {
if (fd == -1) {
return;
}
// Buffered JIT symbols.
void JITSymbols::Register(Core::JITSymbolBuffer *Buffer, const void *HostAddr, uint64_t GuestAddr, uint32_t CodeSize) {
if (fd == -1) return;
// Calculate remaining sizes.
const auto RemainingSize = Buffer->BUFFER_SIZE - Buffer->Offset;
const auto CurrentBufferOffset = &Buffer->Buffer[Buffer->Offset];
// Calculate remaining sizes.
const auto RemainingSize = Buffer->BUFFER_SIZE - Buffer->Offset;
const auto CurrentBufferOffset = &Buffer->Buffer[Buffer->Offset];
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto FMTResult = fmt::format_to_n(CurrentBufferOffset, RemainingSize, "{} {:x} JIT_0x{:x}_{}\n", HostAddr, CodeSize, GuestAddr, HostAddr);
if (FMTResult.out >= &Buffer->Buffer[Buffer->BUFFER_SIZE]) {
// Couldn't fit, need to force a write.
WriteBuffer(Buffer, true);
// Rerun
Register(Buffer, HostAddr, GuestAddr, CodeSize);
return;
}
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto FMTResult = fmt::format_to_n(CurrentBufferOffset, RemainingSize, "{} {:x} JIT_0x{:x}_{}\n", HostAddr, CodeSize, GuestAddr, HostAddr);
if (FMTResult.out >= &Buffer->Buffer[Buffer->BUFFER_SIZE]) {
// Couldn't fit, need to force a write.
WriteBuffer(Buffer, true);
// Rerun
Register(Buffer, HostAddr, GuestAddr, CodeSize);
Buffer->Offset += FMTResult.size;
WriteBuffer(Buffer);
}
void JITSymbols::Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset) {
if (fd == -1) {
return;
}
// Calculate remaining sizes.
const auto RemainingSize = Buffer->BUFFER_SIZE - Buffer->Offset;
const auto CurrentBufferOffset = &Buffer->Buffer[Buffer->Offset];
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto FMTResult =
fmt::format_to_n(CurrentBufferOffset, RemainingSize, "{} {:x} {}+0x{:x} ({})\n", HostAddr, CodeSize, Name, Offset, HostAddr);
if (FMTResult.out >= &Buffer->Buffer[Buffer->BUFFER_SIZE]) {
// Couldn't fit, need to force a write.
WriteBuffer(Buffer, true);
// Rerun
Register(Buffer, HostAddr, CodeSize, Name, Offset);
return;
}
Buffer->Offset += FMTResult.size;
WriteBuffer(Buffer);
}
void JITSymbols::RegisterNamedRegion(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name) {
if (fd == -1) {
return;
}
// Calculate remaining sizes.
const auto RemainingSize = Buffer->BUFFER_SIZE - Buffer->Offset;
const auto CurrentBufferOffset = &Buffer->Buffer[Buffer->Offset];
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto FMTResult = fmt::format_to_n(CurrentBufferOffset, RemainingSize, "{} {:x} {}\n", HostAddr, CodeSize, Name);
if (FMTResult.out >= &Buffer->Buffer[Buffer->BUFFER_SIZE]) {
// Couldn't fit, need to force a write.
WriteBuffer(Buffer, true);
// Rerun
RegisterNamedRegion(Buffer, HostAddr, CodeSize, Name);
return;
}
Buffer->Offset += FMTResult.size;
WriteBuffer(Buffer);
}
void JITSymbols::WriteBuffer(Core::JITSymbolBuffer* Buffer, bool ForceWrite) {
auto Now = std::chrono::steady_clock::now();
if (!ForceWrite) {
if (((Buffer->LastWrite - Now) < Buffer->MAXIMUM_THRESHOLD) && Buffer->Offset < Buffer->NEEDS_WRITE_DISTANCE) {
// Still buffering, no need to write.
return;
}
Buffer->Offset += FMTResult.size;
WriteBuffer(Buffer);
}
void JITSymbols::Register(Core::JITSymbolBuffer *Buffer, const void *HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset) {
if (fd == -1) return;
// Calculate remaining sizes.
const auto RemainingSize = Buffer->BUFFER_SIZE - Buffer->Offset;
const auto CurrentBufferOffset = &Buffer->Buffer[Buffer->Offset];
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto FMTResult = fmt::format_to_n(CurrentBufferOffset, RemainingSize, "{} {:x} {}+0x{:x} ({})\n", HostAddr, CodeSize, Name, Offset, HostAddr);
if (FMTResult.out >= &Buffer->Buffer[Buffer->BUFFER_SIZE]) {
// Couldn't fit, need to force a write.
WriteBuffer(Buffer, true);
// Rerun
Register(Buffer, HostAddr, CodeSize, Name, Offset);
return;
}
Buffer->Offset += FMTResult.size;
WriteBuffer(Buffer);
Buffer->LastWrite = Now;
auto Result = write(fd, Buffer->Buffer, Buffer->Offset);
if (Result == -1 && errno == EBADF) {
fd = -1;
}
void JITSymbols::RegisterNamedRegion(Core::JITSymbolBuffer *Buffer, const void *HostAddr, uint32_t CodeSize, std::string_view Name) {
if (fd == -1) return;
// Calculate remaining sizes.
const auto RemainingSize = Buffer->BUFFER_SIZE - Buffer->Offset;
const auto CurrentBufferOffset = &Buffer->Buffer[Buffer->Offset];
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
const auto FMTResult = fmt::format_to_n(CurrentBufferOffset, RemainingSize, "{} {:x} {}\n", HostAddr, CodeSize, Name);
if (FMTResult.out >= &Buffer->Buffer[Buffer->BUFFER_SIZE]) {
// Couldn't fit, need to force a write.
WriteBuffer(Buffer, true);
// Rerun
RegisterNamedRegion(Buffer, HostAddr, CodeSize, Name);
return;
}
Buffer->Offset += FMTResult.size;
WriteBuffer(Buffer);
}
void JITSymbols::WriteBuffer(Core::JITSymbolBuffer *Buffer, bool ForceWrite) {
auto Now = std::chrono::steady_clock::now();
if (!ForceWrite) {
if (((Buffer->LastWrite - Now) < Buffer->MAXIMUM_THRESHOLD) &&
Buffer->Offset < Buffer->NEEDS_WRITE_DISTANCE) {
// Still buffering, no need to write.
return;
}
}
Buffer->LastWrite = Now;
auto Result = write(fd, Buffer->Buffer, Buffer->Offset);
if (Result == -1 && errno == EBADF) {
fd = -1;
}
Buffer->Offset = 0;
}
Buffer->Offset = 0;
}
} // namespace FEXCore
+8 -8
View File
@@ -17,20 +17,20 @@ public:
~JITSymbols();
void InitFile();
void RegisterNamedRegion(const void *HostAddr, uint32_t CodeSize, std::string_view Name);
void RegisterJITSpace(const void *HostAddr, uint32_t CodeSize);
void RegisterNamedRegion(const void* HostAddr, uint32_t CodeSize, std::string_view Name);
void RegisterJITSpace(const void* HostAddr, uint32_t CodeSize);
// Allocate JIT buffer.
static fextl::unique_ptr<Core::JITSymbolBuffer> AllocateBuffer() {
return fextl::make_unique<Core::JITSymbolBuffer>();
}
void Register(Core::JITSymbolBuffer *Buffer, const void *HostAddr, uint64_t GuestAddr, uint32_t CodeSize);
void Register(Core::JITSymbolBuffer *Buffer, const void *HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset);
void RegisterNamedRegion(Core::JITSymbolBuffer *Buffer, const void *HostAddr, uint32_t CodeSize, std::string_view Name);
void Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint64_t GuestAddr, uint32_t CodeSize);
void Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset);
void RegisterNamedRegion(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name);
private:
int fd{-1};
void WriteBuffer(Core::JITSymbolBuffer *Buffer, bool ForceWrite = false);
int fd {-1};
void WriteBuffer(Core::JITSymbolBuffer* Buffer, bool ForceWrite = false);
};
}
} // namespace FEXCore
+93 -123
View File
@@ -1,10 +1,10 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <FEXCore/Utils/BitUtils.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/fextl/sstream.h>
#include <FEXCore/fextl/string.h>
#include <FEXHeaderUtils/BitUtils.h>
#include <cmath>
#include <cstring>
@@ -45,13 +45,13 @@ struct FEX_PACKED X80SoftFloat {
uint16_t Exponent : 15;
uint16_t Sign : 1;
X80SoftFloat() { memset(this, 0, sizeof(*this)); }
X80SoftFloat() {
memset(this, 0, sizeof(*this));
}
X80SoftFloat(uint16_t _Sign, uint16_t _Exponent, uint64_t _Significand)
: Significand {_Significand}
, Exponent {_Exponent}
, Sign {_Sign}
{
}
, Sign {_Sign} {}
fextl::string str() const {
fextl::ostringstream string;
@@ -63,21 +63,19 @@ struct FEX_PACKED X80SoftFloat {
}
// Ops
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FADD(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FADD(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
faddp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -85,21 +83,19 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FSUB(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FSUB(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fsubp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -107,21 +103,19 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FMUL(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FMUL(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fmulp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -129,21 +123,19 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FDIV(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FDIV(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fdivp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -151,11 +143,10 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FREM(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FREM(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
@@ -163,10 +154,9 @@ struct FEX_PACKED X80SoftFloat {
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -174,11 +164,10 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FREM1(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FREM1(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
@@ -186,10 +175,9 @@ struct FEX_PACKED X80SoftFloat {
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -197,30 +185,27 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FRNDINT(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FRNDINT(const X80SoftFloat& lhs) {
return extF80_roundToInt(lhs, softfloat_roundingMode, false);
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FRNDINT(X80SoftFloat const &lhs, uint_fast8_t RoundMode) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FRNDINT(const X80SoftFloat& lhs, uint_fast8_t RoundMode) {
return extF80_roundToInt(lhs, RoundMode, false);
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FXTRACT_SIG(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FXTRACT_SIG(const X80SoftFloat& lhs) {
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs]; # st0
fxtract;
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs)
: "st", "st(1)");
return Result;
#else
@@ -231,20 +216,19 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FXTRACT_EXP(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FXTRACT_EXP(const X80SoftFloat& lhs) {
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs]; # st0
fxtract;
ffreep %%st(0);
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs)
: "st", "st(1)");
return Result;
#else
@@ -253,19 +237,17 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static void FCMP(X80SoftFloat const &lhs, X80SoftFloat const &rhs, bool *eq, bool *lt, bool *nan) {
FEXCORE_PRESERVE_ALL_ATTR static void FCMP(const X80SoftFloat& lhs, const X80SoftFloat& rhs, bool* eq, bool* lt, bool* nan) {
*eq = extF80_eq(lhs, rhs);
*lt = extF80_lt(lhs, rhs);
*nan = IsNan(lhs) || IsNan(rhs);
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FSCALE(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FSCALE(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
WARN_ONCE_FMT("x87: Application used FSCALE which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
@@ -273,10 +255,9 @@ struct FEX_PACKED X80SoftFloat {
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -289,20 +270,19 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat F2XM1(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat F2XM1(const X80SoftFloat& lhs) {
WARN_ONCE_FMT("x87: Application used F2XM1 which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs]; # st0
f2xm1; # st0 = 2^st(0) - 1
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
: [result] "=m"(Result)
: [lhs] "m"(lhs)
: "st");
return Result;
#else
@@ -313,22 +293,20 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FYL2X(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FYL2X(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
WARN_ONCE_FMT("x87: Application used FYL2X which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[rhs]; # st(1)
fldt %[lhs]; # st(0)
fyl2x; # st(1) * log2l(st(0))
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -339,22 +317,20 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FATAN(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FATAN(const X80SoftFloat& lhs, const X80SoftFloat& rhs) {
WARN_ONCE_FMT("x87: Application used FATAN which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs];
fldt %[rhs];
fpatan;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
: [result] "=m"(Result)
: [lhs] "m"(lhs), [rhs] "m"(rhs)
: "st", "st(1)");
return Result;
#else
@@ -365,21 +341,20 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FTAN(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FTAN(const X80SoftFloat& lhs) {
WARN_ONCE_FMT("x87: Application used FTAN which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs]; # st0
fptan;
ffreep %%st(0);
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
: [result] "=m"(Result)
: [lhs] "m"(lhs)
: "st");
return Result;
#else
@@ -389,20 +364,19 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FSIN(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FSIN(const X80SoftFloat& lhs) {
WARN_ONCE_FMT("x87: Application used FSIN which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs]; # st0
fsin;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
: [result] "=m"(Result)
: [lhs] "m"(lhs)
: "st");
return Result;
#else
@@ -412,20 +386,19 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FCOS(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FCOS(const X80SoftFloat& lhs) {
WARN_ONCE_FMT("x87: Application used FCOS which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs]; # st0
fcos;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
: [result] "=m"(Result)
: [lhs] "m"(lhs)
: "st");
return Result;
#else
@@ -435,19 +408,18 @@ struct FEX_PACKED X80SoftFloat {
#endif
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat FSQRT(X80SoftFloat const &lhs) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat FSQRT(const X80SoftFloat& lhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
asm(R"(
fninit;
fldt %[lhs]; # st0
fsqrt;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
: [result] "=m"(Result)
: [lhs] "m"(lhs)
: "st");
return Result;
#else
@@ -471,7 +443,7 @@ struct FEX_PACKED X80SoftFloat {
const float128_t Result = extF80_to_f128(*this);
return FEXCore::BitCast<BIGFLOAT>(Result);
#else
BIGFLOAT result{};
BIGFLOAT result {};
memcpy(&result, this, sizeof(result));
return result;
#endif
@@ -570,19 +542,17 @@ struct FEX_PACKED X80SoftFloat {
}
operator extFloat80_t() const {
extFloat80_t Result{};
extFloat80_t Result {};
Result.signif = Significand;
Result.signExp = Exponent | (Sign << 15);
return Result;
}
static bool IsNan(X80SoftFloat const &lhs) {
return (lhs.Exponent == 0x7FFF) &&
(lhs.Significand & IntegerBit) &&
(lhs.Significand & Bottom62Significand);
static bool IsNan(const X80SoftFloat& lhs) {
return (lhs.Exponent == 0x7FFF) && (lhs.Significand & IntegerBit) && (lhs.Significand & Bottom62Significand);
}
static bool SignBit(X80SoftFloat const &lhs) {
static bool SignBit(const X80SoftFloat& lhs) {
return lhs.Sign;
}
+47 -40
View File
@@ -7,44 +7,51 @@
#include <optional>
namespace FEXCore::StrConv {
[[maybe_unused]] static bool Conv(std::string_view Value, bool *Result) {
*Result = std::strtoull(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]] static bool Conv(std::string_view Value, uint8_t *Result) {
*Result = std::strtoul(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]] static bool Conv(std::string_view Value, uint16_t *Result) {
*Result = std::strtoul(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]] static bool Conv(std::string_view Value, uint32_t *Result) {
*Result = std::strtoul(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]] static bool Conv(std::string_view Value, int32_t *Result) {
*Result = std::strtol(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]] static bool Conv(std::string_view Value, uint64_t *Result) {
*Result = std::strtoull(Value.data(), nullptr, 0);
return true;
}
template <typename T,
typename = std::enable_if<std::is_enum<T>::value, T>>
[[maybe_unused]] static bool Conv(std::string_view Value, T *Result) {
*Result = static_cast<T>(std::stoull(Value.data(), nullptr, 0));
return true;
}
[[maybe_unused]] static bool Conv(std::string_view Value, fextl::string *Result) {
*Result = Value;
return true;
}
[[maybe_unused]]
static bool Conv(std::string_view Value, bool* Result) {
*Result = std::strtoull(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]]
static bool Conv(std::string_view Value, uint8_t* Result) {
*Result = std::strtoul(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]]
static bool Conv(std::string_view Value, uint16_t* Result) {
*Result = std::strtoul(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]]
static bool Conv(std::string_view Value, uint32_t* Result) {
*Result = std::strtoul(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]]
static bool Conv(std::string_view Value, int32_t* Result) {
*Result = std::strtol(Value.data(), nullptr, 0);
return true;
}
[[maybe_unused]]
static bool Conv(std::string_view Value, uint64_t* Result) {
*Result = std::strtoull(Value.data(), nullptr, 0);
return true;
}
template<typename T, typename = std::enable_if<std::is_enum<T>::value, T>>
[[maybe_unused]]
static bool Conv(std::string_view Value, T* Result) {
*Result = static_cast<T>(std::stoull(Value.data(), nullptr, 0));
return true;
}
[[maybe_unused]]
static bool Conv(std::string_view Value, fextl::string* Result) {
*Result = Value;
return true;
}
} // namespace FEXCore::StrConv
-31
View File
@@ -1,31 +0,0 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <FEXCore/fextl/string.h>
namespace FEXCore::StringUtils {
// Trim the left side of the string of whitespace and new lines
[[maybe_unused]] static fextl::string LeftTrim(fextl::string String, std::string_view TrimTokens = " \t\n\r") {
size_t pos = fextl::string::npos;
if ((pos = String.find_first_not_of(TrimTokens)) != fextl::string::npos) {
String.erase(0, pos);
}
return String;
}
// Trim the right side of the string of whitespace and new lines
[[maybe_unused]] static fextl::string RightTrim(fextl::string String, std::string_view TrimTokens = " \t\n\r") {
size_t pos = fextl::string::npos;
if ((pos = String.find_last_not_of(TrimTokens)) != fextl::string::npos) {
String.erase(String.begin() + pos + 1, String.end());
}
return String;
}
// Trim both the left and right of the string of whitespace and new lines
[[maybe_unused]] static fextl::string Trim(fextl::string String, std::string_view TrimTokens = " \t\n\r") {
return RightTrim(LeftTrim(std::move(String), TrimTokens), TrimTokens);
}
}
+408 -426
View File
@@ -1,6 +1,5 @@
// SPDX-License-Identifier: MIT
#include "Common/StringConv.h"
#include "Common/StringUtils.h"
#include "FEXCore/Utils/EnumUtils.h"
#include <FEXCore/Config/Config.h>
@@ -8,6 +7,7 @@
#include <FEXCore/Utils/CPUInfo.h>
#include <FEXCore/Utils/FileLoading.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/StringUtils.h>
#include <FEXCore/fextl/fmt.h>
#include <FEXCore/fextl/list.h>
#include <FEXCore/fextl/map.h>
@@ -29,7 +29,7 @@
#include <utility>
namespace FEXCore::Context {
class Context;
class Context;
}
namespace FEXCore::Config {
@@ -40,482 +40,464 @@ namespace DefaultValues {
#define OPT_STRARRAY(group, enum, json, default) OPT_STR(group, enum, json, default)
#define OPT_STRENUM(group, enum, json, default) const uint64_t P(enum) = FEXCore::ToUnderlying(P(default));
#include <FEXCore/Config/ConfigValues.inl>
} // namespace DefaultValues
enum Paths {
PATH_DATA_DIR = 0,
PATH_CONFIG_DIR_LOCAL,
PATH_CONFIG_DIR_GLOBAL,
PATH_CONFIG_FILE_LOCAL,
PATH_CONFIG_FILE_GLOBAL,
PATH_CONFIG_TELEMETRY_FOLDER,
PATH_LAST,
};
static std::array<fextl::string, Paths::PATH_LAST> Paths;
void SetDataDirectory(const std::string_view Path) {
Paths[PATH_DATA_DIR] = Path;
}
enum Paths {
PATH_DATA_DIR = 0,
PATH_CONFIG_DIR_LOCAL,
PATH_CONFIG_DIR_GLOBAL,
PATH_CONFIG_FILE_LOCAL,
PATH_CONFIG_FILE_GLOBAL,
PATH_LAST,
};
static std::array<fextl::string, Paths::PATH_LAST> Paths;
void SetConfigDirectory(const std::string_view Path, bool Global) {
Paths[PATH_CONFIG_DIR_LOCAL + Global] = Path;
}
void SetDataDirectory(const std::string_view Path) {
Paths[PATH_DATA_DIR] = Path;
}
void SetConfigFileLocation(const std::string_view Path, bool Global) {
Paths[PATH_CONFIG_FILE_LOCAL + Global] = Path;
}
void SetConfigDirectory(const std::string_view Path, bool Global) {
Paths[PATH_CONFIG_DIR_LOCAL + Global] = Path;
}
void SetConfigFileLocation(const std::string_view Path, bool Global) {
Paths[PATH_CONFIG_FILE_LOCAL + Global] = Path;
}
fextl::string const& GetDataDirectory() {
return Paths[PATH_DATA_DIR];
}
fextl::string const& GetConfigDirectory(bool Global) {
return Paths[PATH_CONFIG_DIR_LOCAL + Global];
}
fextl::string const& GetConfigFileLocation(bool Global) {
return Paths[PATH_CONFIG_FILE_LOCAL + Global];
}
fextl::string GetApplicationConfig(const std::string_view Program, bool Global) {
fextl::string ConfigFile = GetConfigDirectory(Global);
if (!Global &&
!FHU::Filesystem::Exists(ConfigFile) &&
!FHU::Filesystem::CreateDirectories(ConfigFile)) {
LogMan::Msg::DFmt("Couldn't create config directory: '{}'", ConfigFile);
// Let's go local in this case
return fextl::fmt::format("./{}.json", Program);
const fextl::string& GetTelemetryDirectory() {
auto& Path = Paths[PATH_CONFIG_TELEMETRY_FOLDER];
if (Path.empty()) {
FEX_CONFIG_OPT(TelemetryDirectory, TELEMETRYDIRECTORY);
if (!TelemetryDirectory().empty()) {
Path = TelemetryDirectory;
Path += "/";
} else {
Path = Config::GetDataDirectory() + "Telemetry/";
}
}
ConfigFile += "AppConfig/";
return Path;
}
// Attempt to create the local folder if it doesn't exist
if (!Global &&
!FHU::Filesystem::Exists(ConfigFile) &&
!FHU::Filesystem::CreateDirectories(ConfigFile)) {
// Let's go local in this case
return fextl::fmt::format("./{}.json", Program);
const fextl::string& GetDataDirectory() {
return Paths[PATH_DATA_DIR];
}
const fextl::string& GetConfigDirectory(bool Global) {
return Paths[PATH_CONFIG_DIR_LOCAL + Global];
}
const fextl::string& GetConfigFileLocation(bool Global) {
return Paths[PATH_CONFIG_FILE_LOCAL + Global];
}
fextl::string GetApplicationConfig(const std::string_view Program, bool Global) {
fextl::string ConfigFile = GetConfigDirectory(Global);
if (!Global && !FHU::Filesystem::Exists(ConfigFile) && !FHU::Filesystem::CreateDirectories(ConfigFile)) {
LogMan::Msg::DFmt("Couldn't create config directory: '{}'", ConfigFile);
// Let's go local in this case
return fextl::fmt::format("./{}.json", Program);
}
ConfigFile += "AppConfig/";
// Attempt to create the local folder if it doesn't exist
if (!Global && !FHU::Filesystem::Exists(ConfigFile) && !FHU::Filesystem::CreateDirectories(ConfigFile)) {
// Let's go local in this case
return fextl::fmt::format("./{}.json", Program);
}
return fextl::fmt::format("{}{}.json", ConfigFile, Program);
}
void SetConfig(FEXCore::Context::Context* CTX, ConfigOption Option, uint64_t Config) {}
void SetConfig(FEXCore::Context::Context* CTX, ConfigOption Option, const fextl::string& Config) {}
uint64_t GetConfig(FEXCore::Context::Context* CTX, ConfigOption Option) {
return 0;
}
static fextl::map<FEXCore::Config::LayerType, fextl::unique_ptr<FEXCore::Config::Layer>> ConfigLayers;
static FEXCore::Config::Layer* Meta {};
constexpr std::array<FEXCore::Config::LayerType, 10> LoadOrder = {
FEXCore::Config::LayerType::LAYER_GLOBAL_MAIN, FEXCore::Config::LayerType::LAYER_MAIN,
FEXCore::Config::LayerType::LAYER_GLOBAL_STEAM_APP, FEXCore::Config::LayerType::LAYER_GLOBAL_APP,
FEXCore::Config::LayerType::LAYER_LOCAL_STEAM_APP, FEXCore::Config::LayerType::LAYER_LOCAL_APP,
FEXCore::Config::LayerType::LAYER_ARGUMENTS, FEXCore::Config::LayerType::LAYER_USER_OVERRIDE,
FEXCore::Config::LayerType::LAYER_ENVIRONMENT, FEXCore::Config::LayerType::LAYER_TOP};
Layer::Layer(const LayerType _Type)
: Type {_Type} {}
Layer::~Layer() {}
class MetaLayer final : public FEXCore::Config::Layer {
public:
MetaLayer(const LayerType _Type)
: FEXCore::Config::Layer(_Type) {}
~MetaLayer() {}
void Load();
private:
void MergeConfigMap(const LayerOptions& Options);
void MergeEnvironmentVariables(const ConfigOption& Option, const LayerValue& Value);
};
void MetaLayer::Load() {
OptionMap.clear();
for (auto CurrentLayer = LoadOrder.begin(); CurrentLayer != LoadOrder.end(); ++CurrentLayer) {
auto it = ConfigLayers.find(*CurrentLayer);
if (it != ConfigLayers.end() && *CurrentLayer != Type) {
// Merge this layer's options to this layer
MergeConfigMap(it->second->GetOptionMap());
}
}
}
return fextl::fmt::format("{}{}.json", ConfigFile, Program);
void MetaLayer::MergeEnvironmentVariables(const ConfigOption& Option, const LayerValue& Value) {
// Environment variables need a bit of additional work
// We want to merge the arrays rather than overwrite entirely
auto MetaEnvironment = OptionMap.find(Option);
if (MetaEnvironment == OptionMap.end()) {
// Doesn't exist, just insert
OptionMap.insert_or_assign(Option, Value);
return;
}
void SetConfig(FEXCore::Context::Context *CTX, ConfigOption Option, uint64_t Config) {
}
// If an environment variable exists in both current meta and in the incoming layer then the meta layer value is overwritten
fextl::unordered_map<fextl::string, fextl::string> LookupMap;
const auto AddToMap = [&LookupMap](const FEXCore::Config::LayerValue& Value) {
for (const auto& EnvVar : Value) {
const auto ItEq = EnvVar.find_first_of('=');
if (ItEq == fextl::string::npos) {
// Broken environment variable
// Skip
continue;
}
auto Key = fextl::string(EnvVar.begin(), EnvVar.begin() + ItEq);
auto Value = fextl::string(EnvVar.begin() + ItEq + 1, EnvVar.end());
void SetConfig(FEXCore::Context::Context *CTX, ConfigOption Option, fextl::string const &Config) {
}
uint64_t GetConfig(FEXCore::Context::Context *CTX, ConfigOption Option) {
return 0;
}
static fextl::map<FEXCore::Config::LayerType, fextl::unique_ptr<FEXCore::Config::Layer>> ConfigLayers;
static FEXCore::Config::Layer *Meta{};
constexpr std::array<FEXCore::Config::LayerType, 9> LoadOrder = {
FEXCore::Config::LayerType::LAYER_GLOBAL_MAIN,
FEXCore::Config::LayerType::LAYER_MAIN,
FEXCore::Config::LayerType::LAYER_GLOBAL_STEAM_APP,
FEXCore::Config::LayerType::LAYER_GLOBAL_APP,
FEXCore::Config::LayerType::LAYER_LOCAL_STEAM_APP,
FEXCore::Config::LayerType::LAYER_LOCAL_APP,
FEXCore::Config::LayerType::LAYER_ARGUMENTS,
FEXCore::Config::LayerType::LAYER_ENVIRONMENT,
FEXCore::Config::LayerType::LAYER_TOP
// Add the key to the map, overwriting whatever previous value was there
LookupMap.insert_or_assign(std::move(Key), std::move(Value));
}
};
Layer::Layer(const LayerType _Type)
: Type {_Type} {
AddToMap(MetaEnvironment->second);
AddToMap(Value);
// Now with the two layers merged in the map
// Add all the values to the option
Erase(Option);
for (auto& Val : LookupMap) {
// Set will emplace multiple options in to its list
Set(Option, Val.first + "=" + Val.second);
}
}
Layer::~Layer() {
}
class MetaLayer final : public FEXCore::Config::Layer {
public:
MetaLayer(const LayerType _Type)
: FEXCore::Config::Layer (_Type) {
}
~MetaLayer() {
}
void Load();
private:
void MergeConfigMap(const LayerOptions &Options);
void MergeEnvironmentVariables(ConfigOption const &Option, LayerValue const &Value);
};
void MetaLayer::Load() {
OptionMap.clear();
for (auto CurrentLayer = LoadOrder.begin(); CurrentLayer != LoadOrder.end(); ++CurrentLayer) {
auto it = ConfigLayers.find(*CurrentLayer);
if (it != ConfigLayers.end() && *CurrentLayer != Type) {
// Merge this layer's options to this layer
MergeConfigMap(it->second->GetOptionMap());
}
void MetaLayer::MergeConfigMap(const LayerOptions& Options) {
// Insert this layer's options, overlaying previous options that exist here
for (auto& it : Options) {
if (it.first == FEXCore::Config::ConfigOption::CONFIG_ENV || it.first == FEXCore::Config::ConfigOption::CONFIG_HOSTENV) {
MergeEnvironmentVariables(it.first, it.second);
} else {
OptionMap.insert_or_assign(it.first, it.second);
}
}
}
void Initialize() {
AddLayer(fextl::make_unique<MetaLayer>(FEXCore::Config::LayerType::LAYER_TOP));
Meta = ConfigLayers.begin()->second.get();
}
void MetaLayer::MergeEnvironmentVariables(ConfigOption const &Option, LayerValue const &Value) {
// Environment variables need a bit of additional work
// We want to merge the arrays rather than overwrite entirely
auto MetaEnvironment = OptionMap.find(Option);
if (MetaEnvironment == OptionMap.end()) {
// Doesn't exist, just insert
OptionMap.insert_or_assign(Option, Value);
return;
}
void Shutdown() {
ConfigLayers.clear();
Meta = nullptr;
}
// If an environment variable exists in both current meta and in the incoming layer then the meta layer value is overwritten
fextl::unordered_map<fextl::string, fextl::string> LookupMap;
const auto AddToMap = [&LookupMap](FEXCore::Config::LayerValue const &Value) {
for (const auto &EnvVar : Value) {
const auto ItEq = EnvVar.find_first_of('=');
if (ItEq == fextl::string::npos) {
// Broken environment variable
// Skip
continue;
}
auto Key = fextl::string(EnvVar.begin(), EnvVar.begin() + ItEq);
auto Value = fextl::string(EnvVar.begin() + ItEq + 1, EnvVar.end());
// Add the key to the map, overwriting whatever previous value was there
LookupMap.insert_or_assign(std::move(Key), std::move(Value));
}
};
AddToMap(MetaEnvironment->second);
AddToMap(Value);
// Now with the two layers merged in the map
// Add all the values to the option
Erase(Option);
for (auto &Val : LookupMap) {
// Set will emplace multiple options in to its list
Set(Option, Val.first + "=" + Val.second);
void Load() {
for (auto CurrentLayer = LoadOrder.begin(); CurrentLayer != LoadOrder.end(); ++CurrentLayer) {
auto it = ConfigLayers.find(*CurrentLayer);
if (it != ConfigLayers.end()) {
it->second->Load();
}
}
}
void MetaLayer::MergeConfigMap(const LayerOptions &Options) {
// Insert this layer's options, overlaying previous options that exist here
for (auto &it : Options) {
if (it.first == FEXCore::Config::ConfigOption::CONFIG_ENV ||
it.first == FEXCore::Config::ConfigOption::CONFIG_HOSTENV) {
MergeEnvironmentVariables(it.first, it.second);
}
else {
OptionMap.insert_or_assign(it.first, it.second);
}
}
}
void Initialize() {
AddLayer(fextl::make_unique<MetaLayer>(FEXCore::Config::LayerType::LAYER_TOP));
Meta = ConfigLayers.begin()->second.get();
}
void Shutdown() {
ConfigLayers.clear();
Meta = nullptr;
}
void Load() {
for (auto CurrentLayer = LoadOrder.begin(); CurrentLayer != LoadOrder.end(); ++CurrentLayer) {
auto it = ConfigLayers.find(*CurrentLayer);
if (it != ConfigLayers.end()) {
it->second->Load();
}
}
}
fextl::string ExpandPath(fextl::string const &ContainerPrefix, fextl::string PathName) {
if (PathName.empty()) {
return {};
}
// Expand home if it exists
if (FHU::Filesystem::IsRelative(PathName)) {
fextl::string Home = getenv("HOME") ?: "";
// Home expansion only works if it is the first character
// This matches bash behaviour
if (PathName.at(0) == '~') {
PathName.replace(0, 1, Home);
return PathName;
}
// Expand relative path to absolute
char ExistsTempPath[PATH_MAX];
char *RealPath = FHU::Filesystem::Absolute(PathName.c_str(), ExistsTempPath);
if (RealPath) {
PathName = RealPath;
}
// Only return if it exists
if (FHU::Filesystem::Exists(PathName)) {
return PathName;
}
}
else {
// If the containerprefix and pathname isn't empty
// Then we check if the pathname exists in our current namespace
// If the path DOESN'T exist but DOES exist with the prefix applied
// then redirect to the prefix
//
// This might not be expected behaviour for some edge cases but since
// all paths aren't mounted inside the container, then it'll be fine
//
// Main catch case for this is the default thunk install folders
// HostThunks: $CMAKE_INSTALL_PREFIX/lib/fex-emu/HostThunks/
// GuestThunks: $CMAKE_INSTALL_PREFIX/share/fex-emu/GuestThunks/
if (!ContainerPrefix.empty() && !PathName.empty()) {
if (!FHU::Filesystem::Exists(PathName)) {
auto ContainerPath = ContainerPrefix + PathName;
if (FHU::Filesystem::Exists(ContainerPath)) {
return ContainerPath;
}
}
}
}
fextl::string ExpandPath(const fextl::string& ContainerPrefix, fextl::string PathName) {
if (PathName.empty()) {
return {};
}
constexpr char ContainerManager[] = "/run/host/container-manager";
fextl::string FindContainer() {
// We only support pressure-vessel at the moment
if (FHU::Filesystem::Exists(ContainerManager)) {
fextl::vector<char> Manager{};
if (FEXCore::FileLoading::LoadFile(Manager, ContainerManager)) {
// Trim the whitespace, may contain a newline
fextl::string ManagerStr = Manager.data();
ManagerStr = FEXCore::StringUtils::Trim(ManagerStr);
return ManagerStr;
}
// Expand home if it exists
if (FHU::Filesystem::IsRelative(PathName)) {
fextl::string Home = getenv("HOME") ?: "";
// Home expansion only works if it is the first character
// This matches bash behaviour
if (PathName.at(0) == '~') {
PathName.replace(0, 1, Home);
return PathName;
}
return {};
}
fextl::string FindContainerPrefix() {
// We only support pressure-vessel at the moment
if (FHU::Filesystem::Exists(ContainerManager)) {
fextl::vector<char> Manager{};
if (FEXCore::FileLoading::LoadFile(Manager, ContainerManager)) {
// Trim the whitespace, may contain a newline
fextl::string ManagerStr = Manager.data();
ManagerStr = FEXCore::StringUtils::Trim(ManagerStr);
if (strncmp(ManagerStr.data(), "pressure-vessel", Manager.size()) == 0) {
// We are running inside of pressure vessel
// Our $CMAKE_INSTALL_PREFIX paths are now inside of /run/host/$CMAKE_INSTALL_PREFIX
return "/run/host/";
// Expand relative path to absolute
char ExistsTempPath[PATH_MAX];
char* RealPath = FHU::Filesystem::Absolute(PathName.c_str(), ExistsTempPath);
if (RealPath) {
PathName = RealPath;
}
// Only return if it exists
if (FHU::Filesystem::Exists(PathName)) {
return PathName;
}
} else {
// If the containerprefix and pathname isn't empty
// Then we check if the pathname exists in our current namespace
// If the path DOESN'T exist but DOES exist with the prefix applied
// then redirect to the prefix
//
// This might not be expected behaviour for some edge cases but since
// all paths aren't mounted inside the container, then it'll be fine
//
// Main catch case for this is the default thunk install folders
// HostThunks: $CMAKE_INSTALL_PREFIX/lib/fex-emu/HostThunks/
// GuestThunks: $CMAKE_INSTALL_PREFIX/share/fex-emu/GuestThunks/
if (!ContainerPrefix.empty() && !PathName.empty()) {
if (!FHU::Filesystem::Exists(PathName)) {
auto ContainerPath = ContainerPrefix + PathName;
if (FHU::Filesystem::Exists(ContainerPath)) {
return ContainerPath;
}
}
}
return {};
}
return {};
}
void ReloadMetaLayer() {
Meta->Load();
constexpr char ContainerManager[] = "/run/host/container-manager";
// Do configuration option fix ups after everything is reloaded
{
// Always fix up the number of threads and create the configuration
// Otherwise the application could receive zero as the number of threads
FEX_CONFIG_OPT(Cores, THREADS);
if (Cores == 0) {
// When the number of emulated CPU cores is zero then auto detect
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THREADS, fextl::fmt::format("{}", FEXCore::CPUInfo::CalculateNumberOfCPUs()));
fextl::string FindContainer() {
// We only support pressure-vessel at the moment
if (FHU::Filesystem::Exists(ContainerManager)) {
fextl::vector<char> Manager {};
if (FEXCore::FileLoading::LoadFile(Manager, ContainerManager)) {
// Trim the whitespace, may contain a newline
fextl::string ManagerStr = Manager.data();
ManagerStr = FEXCore::StringUtils::Trim(ManagerStr);
return ManagerStr;
}
}
return {};
}
fextl::string FindContainerPrefix() {
// We only support pressure-vessel at the moment
if (FHU::Filesystem::Exists(ContainerManager)) {
fextl::vector<char> Manager {};
if (FEXCore::FileLoading::LoadFile(Manager, ContainerManager)) {
// Trim the whitespace, may contain a newline
fextl::string ManagerStr = Manager.data();
ManagerStr = FEXCore::StringUtils::Trim(ManagerStr);
if (strncmp(ManagerStr.data(), "pressure-vessel", Manager.size()) == 0) {
// We are running inside of pressure vessel
// Our $CMAKE_INSTALL_PREFIX paths are now inside of /run/host/$CMAKE_INSTALL_PREFIX
return "/run/host/";
}
}
}
return {};
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_CORE)) {
// Sanitize Core option
FEX_CONFIG_OPT(Core, CORE);
void ReloadMetaLayer() {
Meta->Load();
// Do configuration option fix ups after everything is reloaded
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_CORE)) {
// Sanitize Core option
FEX_CONFIG_OPT(Core, CORE);
#if (_M_X86_64)
constexpr uint32_t MaxCoreNumber = 1;
constexpr uint32_t MaxCoreNumber = 1;
#else
constexpr uint32_t MaxCoreNumber = 0;
constexpr uint32_t MaxCoreNumber = 0;
#endif
if (Core > MaxCoreNumber) {
// Sanitize the core option by setting the core to the JIT if invalid
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_CORE, fextl::fmt::format("{}", static_cast<uint32_t>(FEXCore::Config::CONFIG_IRJIT)));
if (Core > MaxCoreNumber) {
// Sanitize the core option by setting the core to the JIT if invalid
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_CORE, fextl::fmt::format("{}", static_cast<uint32_t>(FEXCore::Config::CONFIG_IRJIT)));
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_CACHEOBJECTCODECOMPILATION)) {
FEX_CONFIG_OPT(CacheObjectCodeCompilation, CACHEOBJECTCODECOMPILATION);
FEX_CONFIG_OPT(Core, CORE);
}
fextl::string ContainerPrefix {FindContainerPrefix()};
auto ExpandPathIfExists = [&ContainerPrefix](FEXCore::Config::ConfigOption Config, fextl::string PathName) {
auto NewPath = ExpandPath(ContainerPrefix, PathName);
if (!NewPath.empty()) {
FEXCore::Config::EraseSet(Config, NewPath);
}
};
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_ROOTFS)) {
FEX_CONFIG_OPT(PathName, ROOTFS);
auto ExpandedString = ExpandPath(ContainerPrefix, PathName());
if (!ExpandedString.empty()) {
// Adjust the path if it ended up being relative
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_ROOTFS, ExpandedString);
} else if (!PathName().empty()) {
// If the filesystem doesn't exist then let's see if it exists in the fex-emu folder
fextl::string NamedRootFS = GetDataDirectory() + "RootFS/" + PathName();
if (FHU::Filesystem::Exists(NamedRootFS)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_ROOTFS, NamedRootFS);
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_CACHEOBJECTCODECOMPILATION)) {
FEX_CONFIG_OPT(CacheObjectCodeCompilation, CACHEOBJECTCODECOMPILATION);
FEX_CONFIG_OPT(Core, CORE);
}
fextl::string ContainerPrefix { FindContainerPrefix() };
auto ExpandPathIfExists = [&ContainerPrefix](FEXCore::Config::ConfigOption Config, fextl::string PathName) {
auto NewPath = ExpandPath(ContainerPrefix, PathName);
if (!NewPath.empty()) {
FEXCore::Config::EraseSet(Config, NewPath);
}
};
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_ROOTFS)) {
FEX_CONFIG_OPT(PathName, ROOTFS);
auto ExpandedString = ExpandPath(ContainerPrefix,PathName());
if (!ExpandedString.empty()) {
// Adjust the path if it ended up being relative
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_ROOTFS, ExpandedString);
}
else if (!PathName().empty()) {
// If the filesystem doesn't exist then let's see if it exists in the fex-emu folder
fextl::string NamedRootFS = GetDataDirectory() + "RootFS/" + PathName();
if (FHU::Filesystem::Exists(NamedRootFS)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_ROOTFS, NamedRootFS);
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKHOSTLIBS)) {
FEX_CONFIG_OPT(PathName, THUNKHOSTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKHOSTLIBS, PathName());
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKGUESTLIBS)) {
FEX_CONFIG_OPT(PathName, THUNKGUESTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKGUESTLIBS, PathName());
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKCONFIG)) {
FEX_CONFIG_OPT(PathName, THUNKCONFIG);
auto ExpandedString = ExpandPath(ContainerPrefix, PathName());
if (!ExpandedString.empty()) {
// Adjust the path if it ended up being relative
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THUNKCONFIG, ExpandedString);
} else if (!PathName().empty()) {
// If the filesystem doesn't exist then let's see if it exists in the fex-emu folder
fextl::string NamedConfig = GetDataDirectory() + "ThunkConfigs/" + PathName();
if (FHU::Filesystem::Exists(NamedConfig)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THUNKCONFIG, NamedConfig);
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKHOSTLIBS)) {
FEX_CONFIG_OPT(PathName, THUNKHOSTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKHOSTLIBS, PathName());
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKGUESTLIBS)) {
FEX_CONFIG_OPT(PathName, THUNKGUESTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKGUESTLIBS, PathName());
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKCONFIG)) {
FEX_CONFIG_OPT(PathName, THUNKCONFIG);
auto ExpandedString = ExpandPath(ContainerPrefix, PathName());
if (!ExpandedString.empty()) {
// Adjust the path if it ended up being relative
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THUNKCONFIG, ExpandedString);
}
else if (!PathName().empty()) {
// If the filesystem doesn't exist then let's see if it exists in the fex-emu folder
fextl::string NamedConfig = GetDataDirectory() + "ThunkConfigs/" + PathName();
if (FHU::Filesystem::Exists(NamedConfig)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THUNKCONFIG, NamedConfig);
}
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_OUTPUTLOG)) {
FEX_CONFIG_OPT(PathName, OUTPUTLOG);
if (PathName() != "stdout" && PathName() != "stderr" && PathName() != "server") {
ExpandPathIfExists(FEXCore::Config::CONFIG_OUTPUTLOG, PathName());
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_DUMPIR) &&
!FEXCore::Config::Exists(FEXCore::Config::CONFIG_PASSMANAGERDUMPIR)) {
// If DumpIR is set but no PassManagerDumpIR configuration is set, then default to `afteropt`
FEX_CONFIG_OPT(PathName, DUMPIR);
if (PathName() != "no") {
EraseSet(FEXCore::Config::ConfigOption::CONFIG_PASSMANAGERDUMPIR, fextl::fmt::format("{}", static_cast<uint64_t>(FEXCore::Config::PassManagerDumpIR::AFTEROPT)));
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_SINGLESTEP)) {
// Single stepping also enforces single instruction size blocks
Set(FEXCore::Config::ConfigOption::CONFIG_MAXINST, "1");
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_OUTPUTLOG)) {
FEX_CONFIG_OPT(PathName, OUTPUTLOG);
if (PathName() != "stdout" && PathName() != "stderr" && PathName() != "server") {
ExpandPathIfExists(FEXCore::Config::CONFIG_OUTPUTLOG, PathName());
}
}
void AddLayer(fextl::unique_ptr<FEXCore::Config::Layer> _Layer) {
ConfigLayers.emplace(_Layer->GetLayerType(), std::move(_Layer));
}
bool Exists(ConfigOption Option) {
return Meta->OptionExists(Option);
}
std::optional<LayerValue*> All(ConfigOption Option) {
return Meta->All(Option);
}
std::optional<fextl::string*> Get(ConfigOption Option) {
return Meta->Get(Option);
}
void Set(ConfigOption Option, std::string_view Data) {
Meta->Set(Option, Data);
}
void Erase(ConfigOption Option) {
Meta->Erase(Option);
}
void EraseSet(ConfigOption Option, std::string_view Data) {
Meta->EraseSet(Option, Data);
}
template<typename T>
T Value<T>::Get(FEXCore::Config::ConfigOption Option) {
T Result;
auto Value = FEXCore::Config::Get(Option);
if (!FEXCore::StrConv::Conv(**Value, &Result)) {
LOGMAN_MSG_A_FMT("Attempted to convert invalid value");
}
return Result;
}
template<typename T>
T Value<T>::GetIfExists(FEXCore::Config::ConfigOption Option, T Default) {
T Result;
auto Value = FEXCore::Config::Get(Option);
if (Value && FEXCore::StrConv::Conv(**Value, &Result)) {
return Result;
}
else {
return Default;
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_DUMPIR) && !FEXCore::Config::Exists(FEXCore::Config::CONFIG_PASSMANAGERDUMPIR)) {
// If DumpIR is set but no PassManagerDumpIR configuration is set, then default to `afteropt`
FEX_CONFIG_OPT(PathName, DUMPIR);
if (PathName() != "no") {
EraseSet(FEXCore::Config::ConfigOption::CONFIG_PASSMANAGERDUMPIR,
fextl::fmt::format("{}", static_cast<uint64_t>(FEXCore::Config::PassManagerDumpIR::AFTEROPT)));
}
}
template<>
fextl::string Value<fextl::string>::GetIfExists(FEXCore::Config::ConfigOption Option, fextl::string Default) {
auto Value = FEXCore::Config::Get(Option);
if (Value) {
return **Value;
}
else {
return Default;
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_SINGLESTEP)) {
// Single stepping also enforces single instruction size blocks
Set(FEXCore::Config::ConfigOption::CONFIG_MAXINST, "1");
}
template<>
fextl::string Value<fextl::string>::GetIfExists(FEXCore::Config::ConfigOption Option, std::string_view Default) {
auto Value = FEXCore::Config::Get(Option);
if (Value) {
return **Value;
}
else {
return fextl::string(Default);
}
}
template bool Value<bool>::GetIfExists(FEXCore::Config::ConfigOption Option, bool Default);
template int8_t Value<int8_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int8_t Default);
template uint8_t Value<uint8_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint8_t Default);
template int16_t Value<int16_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int16_t Default);
template uint16_t Value<uint16_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint16_t Default);
template int32_t Value<int32_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int32_t Default);
template uint32_t Value<uint32_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint32_t Default);
template int64_t Value<int64_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int64_t Default);
template uint64_t Value<uint64_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint64_t Default);
// Constructor
template Value<fextl::string>::Value(FEXCore::Config::ConfigOption _Option, fextl::string Default);
template Value<bool>::Value(FEXCore::Config::ConfigOption _Option, bool Default);
template Value<uint8_t>::Value(FEXCore::Config::ConfigOption _Option, uint8_t Default);
template Value<uint64_t>::Value(FEXCore::Config::ConfigOption _Option, uint64_t Default);
template<typename T>
void Value<T>::GetListIfExists(FEXCore::Config::ConfigOption Option, fextl::list<fextl::string> *List) {
auto Value = FEXCore::Config::All(Option);
List->clear();
if (Value) {
*List = **Value;
}
}
template void Value<fextl::string>::GetListIfExists(FEXCore::Config::ConfigOption Option, fextl::list<fextl::string> *List);
}
void AddLayer(fextl::unique_ptr<FEXCore::Config::Layer> _Layer) {
ConfigLayers.emplace(_Layer->GetLayerType(), std::move(_Layer));
}
bool Exists(ConfigOption Option) {
return Meta->OptionExists(Option);
}
std::optional<LayerValue*> All(ConfigOption Option) {
return Meta->All(Option);
}
std::optional<fextl::string*> Get(ConfigOption Option) {
return Meta->Get(Option);
}
void Set(ConfigOption Option, std::string_view Data) {
Meta->Set(Option, Data);
}
void Erase(ConfigOption Option) {
Meta->Erase(Option);
}
void EraseSet(ConfigOption Option, std::string_view Data) {
Meta->EraseSet(Option, Data);
}
template<typename T>
T Value<T>::Get(FEXCore::Config::ConfigOption Option) {
T Result;
auto Value = FEXCore::Config::Get(Option);
if (!FEXCore::StrConv::Conv(**Value, &Result)) {
LOGMAN_MSG_A_FMT("Attempted to convert invalid value");
}
return Result;
}
template<typename T>
T Value<T>::GetIfExists(FEXCore::Config::ConfigOption Option, T Default) {
T Result;
auto Value = FEXCore::Config::Get(Option);
if (Value && FEXCore::StrConv::Conv(**Value, &Result)) {
return Result;
} else {
return Default;
}
}
template<>
fextl::string Value<fextl::string>::GetIfExists(FEXCore::Config::ConfigOption Option, fextl::string Default) {
auto Value = FEXCore::Config::Get(Option);
if (Value) {
return **Value;
} else {
return Default;
}
}
template<>
fextl::string Value<fextl::string>::GetIfExists(FEXCore::Config::ConfigOption Option, std::string_view Default) {
auto Value = FEXCore::Config::Get(Option);
if (Value) {
return **Value;
} else {
return fextl::string(Default);
}
}
template bool Value<bool>::GetIfExists(FEXCore::Config::ConfigOption Option, bool Default);
template int8_t Value<int8_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int8_t Default);
template uint8_t Value<uint8_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint8_t Default);
template int16_t Value<int16_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int16_t Default);
template uint16_t Value<uint16_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint16_t Default);
template int32_t Value<int32_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int32_t Default);
template uint32_t Value<uint32_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint32_t Default);
template int64_t Value<int64_t>::GetIfExists(FEXCore::Config::ConfigOption Option, int64_t Default);
template uint64_t Value<uint64_t>::GetIfExists(FEXCore::Config::ConfigOption Option, uint64_t Default);
// Constructor
template Value<fextl::string>::Value(FEXCore::Config::ConfigOption _Option, fextl::string Default);
template Value<bool>::Value(FEXCore::Config::ConfigOption _Option, bool Default);
template Value<uint8_t>::Value(FEXCore::Config::ConfigOption _Option, uint8_t Default);
template Value<uint64_t>::Value(FEXCore::Config::ConfigOption _Option, uint64_t Default);
template<typename T>
void Value<T>::GetListIfExists(FEXCore::Config::ConfigOption Option, fextl::list<fextl::string>* List) {
auto Value = FEXCore::Config::All(Option);
List->clear();
if (Value) {
*List = **Value;
}
}
template void Value<fextl::string>::GetListIfExists(FEXCore::Config::ConfigOption Option, fextl::list<fextl::string>* List);
} // namespace FEXCore::Config
+72 -34
View File
@@ -31,15 +31,6 @@
"Maximum number of instruction to store in a block"
]
},
"Threads": {
"Type": "uint32",
"Default": "0",
"ShortArg": "T",
"Desc": [
"Number of physical hardware threads to tell the process we have.",
"0 will auto detect."
]
},
"CacheObjectCodeCompilation": {
"Type": "uint32",
"Default": "FEXCore::Config::ConfigObjectCodeHandler::CONFIG_NONE",
@@ -59,8 +50,6 @@
"DISABLESVE": "disablesve",
"ENABLEAVX": "enableavx",
"DISABLEAVX": "disableavx",
"ENABLEAVX2": "enableavx2",
"DISABLEAVX2": "disableavx2",
"ENABLEAFP": "enableafp",
"DISABLEAFP": "disableafp",
"ENABLELRCPC": "enablelrcpc",
@@ -82,14 +71,19 @@
"ENABLEFLAGM": "enableflagm",
"DISABLEFLAGM": "disableflagm",
"ENABLEFLAGM2": "enableflagm2",
"DISABLEFLAGM2": "disableflagm2"
"DISABLEFLAGM2": "disableflagm2",
"ENABLECRYPTO": "enablecrypto",
"DISABLECRYPTO": "disablecrypto",
"ENABLERPRES": "enablerpres",
"DISABLERPRES": "disablerpres",
"ENABLEPRESERVEALLABI": "enablepreserveallabi",
"DISABLEPRESERVEALLABI": "disablepreserveallabi"
},
"Desc": [
"Allows controlling of the CPU features in the JIT.",
"\toff: Default CPU features queried from CPU features",
"\t{enable,disable}sve: Will force enable or disable sve even if the host doesn't support it",
"\t{enable,disable}avx: Will force enable or disable avx even if the host doesn't support it",
"\t{enable,disable}avx2: Will force enable or disable avx2 even if the host doesn't support it",
"\t{enable,disable}afp: Will force enable or disable afp even if the host doesn't support it",
"\t{enable,disable}lrcpc: Will force enable or disable lrcpc even if the host doesn't support it",
"\t{enable,disable}lrcpc2: Will force enable or disable lrcpc2 even if the host doesn't support it",
@@ -100,7 +94,30 @@
"\t{enable,disable}atomics: Will force enable or disable ARMv8.1 LSE atomics even if the host doesn't support it",
"\t{enable,disable}fcma: Will force enable or disable fcma even if the host doesn't support it",
"\t{enable,disable}flagm: Will force enable or disable flagm even if the host doesn't support it",
"\t{enable,disable}flagm2: Will force enable or disable flagm2 even if the host doesn't support it"
"\t{enable,disable}flagm2: Will force enable or disable flagm2 even if the host doesn't support it",
"\t{enable,disable}crypto: Will force enable or disable crypto extensions even if the host doesn't support it",
"\t{enable,disable}rpres: Will force enable or disable rpres even if the host doesn't support it",
"\t{enable,disable}preserveallabi: Will force enable or disable preserve_all abi even if the host doesn't support it"
]
},
"CPUID": {
"Type": "strenum",
"Default": "FEXCore::Config::CPUID::OFF",
"Enums": {
"ENABLESHA": "enablesha",
"DISABLESHA": "disablesha"
},
"Desc": [
"Allows controlling of the CPU features are exposed in CPUID.",
"\toff: Default CPU features queried from CPU features",
"\t{enable,disable}sha: Will force enable or disable sha even if the host doesn't support it"
]
},
"SmallTSCScale": {
"Type": "bool",
"Default": "true",
"Desc": [
"Scales the cycle counter on systems that have low frequencies."
]
}
},
@@ -250,23 +267,6 @@
"Disables optimizations passes for debugging."
]
},
"SRA": {
"Type": "bool",
"Default": "true",
"Desc": [
"Set to false to disable Static Register Allocation"
]
},
"Force32BitAllocator": {
"Type": "bool",
"Default": "false",
"Desc": [
"Forces use of the 32-bit allocator on 32-bit applications",
"Used to work around ulimit problems of CI runner",
"Potentially useful for debugging memory problems",
"32-bit allocator is always used if your host kernel is older than 4.17"
]
},
"GlobalJITNaming": {
"Type": "bool",
"Default": "false",
@@ -365,6 +365,14 @@
"File to write FEX output to.",
"[stdout, stderr, server, <Filename>]"
]
},
"TelemetryDirectory": {
"Type": "str",
"Default": "",
"Desc": [
"Redirects the telemetry folder that FEX usually writes to.",
"By default telemetry data is stored in {$FEX_APP_DATA_LOCATION,{$XDG_DATA_HOME,$HOME}/.fex-emu/Telemetry/}"
]
}
},
"Hacks": {
@@ -376,9 +384,8 @@
"Desc": [
"Checks code for modification before execution.",
"\tnone: No checks",
"\tmtrack: Page tracking based invalidation",
"\tfull: Validate code before every run (slow)",
"\tmman: Invalidate on mmap, mprotect, munmap (deprecated, use mtrack)"
"\tmtrack: Page tracking based invalidation (default)",
"\tfull: Validate code before every run (slow)"
]
},
"TSOEnabled": {
@@ -389,6 +396,29 @@
"Highly likely to break any multithreaded application if disabled."
]
},
"VectorTSOEnabled": {
"Type": "bool",
"Default": "false",
"Desc": [
"When TSO emulation is enabled, controls if vector loadstores should also be atomic."
]
},
"MemcpySetTSOEnabled": {
"Type": "bool",
"Default": "false",
"Desc": [
"When TSO emulation is enabled, controls if memcpy and memset should also be atomic.",
"Only affects REP MOVS and REP STOS instructions"
]
},
"HalfBarrierTSOEnabled": {
"Type": "bool",
"Default": "true",
"Desc": [
"When TSO emulation is enabled, controls if unaligned loads and stores should be backpatched to half-barrier atomics.",
"Can be dangerous due to aligned loadstores through the same code now become non-atomic."
]
},
"TSOAutoMigration": {
"Type": "bool",
"Default": "true",
@@ -436,6 +466,14 @@
"Hides the hypervisor CPUID bit when set.",
"Should only be used for applications that have issues with this set."
]
},
"StartupSleep": {
"Type": "uint32",
"Default": "0",
"Desc": [
"Sleeps the process at startup for a duration of seconds.",
"Useful if an application crashes too quickly to attach a debugger."
]
}
},
"Misc": {
+57 -79
View File
@@ -12,84 +12,62 @@
#include <string.h>
#include <utility>
namespace FEXCore::HLE {
class SyscallVisitor;
}
namespace FEXCore::Context {
void InitializeStaticTables(OperatingMode Mode) {
X86Tables::InitializeInfoTables(Mode);
IR::InstallOpcodeHandlers(Mode);
}
fextl::unique_ptr<FEXCore::Context::Context> FEXCore::Context::Context::CreateNewContext() {
return fextl::make_unique<FEXCore::Context::ContextImpl>();
}
bool FEXCore::Context::ContextImpl::InitializeContext() {
// This should be used for generating things that are shared between threads
CPUID.Init(this);
return true;
}
void FEXCore::Context::ContextImpl::SetExitHandler(ExitHandler handler) {
CustomExitHandler = std::move(handler);
}
ExitHandler FEXCore::Context::ContextImpl::GetExitHandler() const {
return CustomExitHandler;
}
void FEXCore::Context::ContextImpl::Stop() {
Stop(false);
}
void FEXCore::Context::ContextImpl::CompileRIP(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP) {
CompileBlock(Thread->CurrentFrame, GuestRIP);
}
FEXCore::Context::ExitReason FEXCore::Context::ContextImpl::GetExitReason() {
return ParentThread->ExitReason;
}
bool FEXCore::Context::ContextImpl::IsDone() const {
return IsPaused();
}
void FEXCore::Context::ContextImpl::GetCPUState(FEXCore::Core::CPUState *State) const {
memcpy(State, ParentThread->CurrentFrame, sizeof(FEXCore::Core::CPUState));
}
void FEXCore::Context::ContextImpl::SetCPUState(const FEXCore::Core::CPUState *State) {
memcpy(ParentThread->CurrentFrame, State, sizeof(FEXCore::Core::CPUState));
}
void FEXCore::Context::ContextImpl::SetCustomCPUBackendFactory(CustomCPUFactoryType Factory) {
CustomCPUFactory = std::move(Factory);
}
HostFeatures FEXCore::Context::ContextImpl::GetHostFeatures() const {
return HostFeatures;
}
void FEXCore::Context::ContextImpl::SetSignalDelegator(FEXCore::SignalDelegator *_SignalDelegation) {
SignalDelegation = _SignalDelegation;
}
void FEXCore::Context::ContextImpl::SetSyscallHandler(FEXCore::HLE::SyscallHandler *Handler) {
SyscallHandler = Handler;
SourcecodeResolver = Handler->GetSourcecodeResolver();
}
FEXCore::CPUID::FunctionResults FEXCore::Context::ContextImpl::RunCPUIDFunction(uint32_t Function, uint32_t Leaf) {
return CPUID.RunFunction(Function, Leaf);
}
FEXCore::CPUID::XCRResults FEXCore::Context::ContextImpl::RunXCRFunction(uint32_t Function) {
return CPUID.RunXCRFunction(Function);
}
FEXCore::CPUID::FunctionResults FEXCore::Context::ContextImpl::RunCPUIDFunctionName(uint32_t Function, uint32_t Leaf, uint32_t CPU) {
return CPUID.RunFunctionName(Function, Leaf, CPU);
}
void InitializeStaticTables(OperatingMode Mode) {
X86Tables::InitializeInfoTables(Mode);
IR::InstallOpcodeHandlers(Mode);
}
fextl::unique_ptr<FEXCore::Context::Context> FEXCore::Context::Context::CreateNewContext() {
return fextl::make_unique<FEXCore::Context::ContextImpl>();
}
void FEXCore::Context::ContextImpl::SetExitHandler(ExitHandler handler) {
CustomExitHandler = std::move(handler);
}
ExitHandler FEXCore::Context::ContextImpl::GetExitHandler() const {
return CustomExitHandler;
}
void FEXCore::Context::ContextImpl::CompileRIP(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP) {
CompileBlock(Thread->CurrentFrame, GuestRIP);
}
void FEXCore::Context::ContextImpl::CompileRIPCount(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP, uint64_t MaxInst) {
CompileBlock(Thread->CurrentFrame, GuestRIP, MaxInst);
}
void FEXCore::Context::ContextImpl::SetCustomCPUBackendFactory(CustomCPUFactoryType Factory) {
CustomCPUFactory = std::move(Factory);
}
HostFeatures FEXCore::Context::ContextImpl::GetHostFeatures() const {
return HostFeatures;
}
void FEXCore::Context::ContextImpl::SetSignalDelegator(FEXCore::SignalDelegator* _SignalDelegation) {
SignalDelegation = _SignalDelegation;
}
void FEXCore::Context::ContextImpl::SetSyscallHandler(FEXCore::HLE::SyscallHandler* Handler) {
SyscallHandler = Handler;
SourcecodeResolver = Handler->GetSourcecodeResolver();
}
FEXCore::CPUID::FunctionResults FEXCore::Context::ContextImpl::RunCPUIDFunction(uint32_t Function, uint32_t Leaf) {
return CPUID.RunFunction(Function, Leaf);
}
FEXCore::CPUID::XCRResults FEXCore::Context::ContextImpl::RunXCRFunction(uint32_t Function) {
return CPUID.RunXCRFunction(Function);
}
FEXCore::CPUID::FunctionResults FEXCore::Context::ContextImpl::RunCPUIDFunctionName(uint32_t Function, uint32_t Leaf, uint32_t CPU) {
return CPUID.RunFunctionName(Function, Leaf, CPU);
}
bool FEXCore::Context::ContextImpl::IsAddressInCodeBuffer(FEXCore::Core::InternalThreadState* Thread, uintptr_t Address) const {
return Thread->CPUBackend->IsAddressInCodeBuffer(Address);
}
} // namespace FEXCore::Context
+314 -356
View File
@@ -13,9 +13,10 @@
#include <FEXCore/Core/HostFeatures.h>
#include <FEXCore/Core/SignalDelegator.h>
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXCore/IR/IR.h>
#include <FEXCore/Utils/CompilerDefs.h>
#include <FEXCore/Utils/DeferredSignalMutex.h>
#include <FEXCore/Utils/Event.h>
#include <FEXCore/Utils/SignalScopeGuards.h>
#include <FEXCore/fextl/memory.h>
#include <FEXCore/fextl/set.h>
#include <FEXCore/fextl/string.h>
@@ -37,7 +38,6 @@
namespace FEXCore {
class CodeLoader;
class ThunkHandler;
class GdbServer;
namespace CodeSerialize {
class CodeObjectSerializeService;
@@ -45,418 +45,376 @@ namespace CodeSerialize {
namespace CPU {
class Arm64JITCore;
class X86JITCore;
class Dispatcher;
}
} // namespace CPU
namespace HLE {
struct SyscallArguments;
class SyscallHandler;
class SourcecodeResolver;
struct SourcecodeMap;
}
}
struct SyscallArguments;
class SyscallHandler;
class SourcecodeResolver;
struct SourcecodeMap;
} // namespace HLE
} // namespace FEXCore
namespace FEXCore::IR {
class RegisterAllocationData;
class IRListView;
class RegisterAllocationData;
struct IRListCopy;
class IRListView;
namespace Validation {
class IRValidation;
}
}
} // namespace FEXCore::IR
namespace FEXCore::Context {
enum CoreRunningMode {
MODE_RUN = 0,
MODE_SINGLESTEP = 1,
};
enum CoreRunningMode {
MODE_RUN = 0,
MODE_SINGLESTEP = 1,
};
class ContextImpl final : public FEXCore::Context::Context {
public:
// Context base class implementation.
bool InitializeContext() override;
struct ExitFunctionLinkData {
uint64_t HostBranch;
uint64_t GuestRIP;
};
FEXCore::Core::InternalThreadState* InitCore(uint64_t InitialRIP, uint64_t StackPointer) override;
using BlockDelinkerFunc = void (*)(FEXCore::Core::CpuStateFrame* Frame, FEXCore::Context::ExitFunctionLinkData* Record);
constexpr uint32_t TSC_SCALE = 128;
constexpr uint32_t TSC_SCALE_MAXIMUM = 1'000'000'000; ///< 1Ghz
void SetExitHandler(ExitHandler handler) override;
ExitHandler GetExitHandler() const override;
class ContextImpl final : public FEXCore::Context::Context {
public:
// Context base class implementation.
bool InitCore() override;
void Pause() override;
void Run() override;
void Stop() override;
void Step() override;
void SetExitHandler(ExitHandler handler) override;
ExitHandler GetExitHandler() const override;
ExitReason RunUntilExit() override;
ExitReason RunUntilExit(FEXCore::Core::InternalThreadState* Thread) override;
void ExecuteThread(FEXCore::Core::InternalThreadState *Thread) override;
void ExecuteThread(FEXCore::Core::InternalThreadState* Thread) override;
void CompileRIP(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP) override;
void CompileRIP(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP) override;
void CompileRIPCount(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP, uint64_t MaxInst) override;
int GetProgramStatus() const override;
void SetCustomCPUBackendFactory(CustomCPUFactoryType Factory) override;
ExitReason GetExitReason() override;
HostFeatures GetHostFeatures() const override;
bool IsDone() const override;
void HandleCallback(FEXCore::Core::InternalThreadState* Thread, uint64_t RIP) override;
void GetCPUState(FEXCore::Core::CPUState *State) const override;
void SetCPUState(const FEXCore::Core::CPUState *State) override;
uint64_t RestoreRIPFromHostPC(FEXCore::Core::InternalThreadState* Thread, uint64_t HostPC) override;
uint32_t ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadState* Thread, bool WasInJIT, uint64_t* HostGPRs, uint64_t PSTATE) override;
void SetFlagsFromCompactedEFLAGS(FEXCore::Core::InternalThreadState* Thread, uint32_t EFLAGS) override;
void SetCustomCPUBackendFactory(CustomCPUFactoryType Factory) override;
void ReconstructXMMRegisters(const FEXCore::Core::InternalThreadState* Thread, __uint128_t* XMM_Low, __uint128_t* YMM_High) override;
void SetXMMRegistersFromState(FEXCore::Core::InternalThreadState* Thread, const __uint128_t* XMM_Low, const __uint128_t* YMM_High) override;
HostFeatures GetHostFeatures() const override;
/**
* @brief Used to create FEX thread objects in preparation for creating a true OS thread. Does set a TID or PID.
*
* @param InitialRIP The starting RIP of this thread
* @param StackPointer The starting RSP of this thread
* @param NewThreadState The initial thread state to setup for our state, if inheriting.
* @param ParentTID The PID that was the parent thread that created this
*
* @return The InternalThreadState object that tracks all of the emulated thread's state
*
* Usecases:
* Parent thread Creation:
* - Thread = CreateThread(InitialRIP, InitialStack, nullptr, 0);
* - CTX->RunUntilExit(Thread);
* OS thread Creation:
* - Thread = CreateThread(0, 0, NewState, PPID);
* - Thread->ExecutionThread = FEXCore::Threads::Thread::Create(ThreadHandler, Arg);
* - ThreadHandler calls `CTX->ExecutionThread(Thread)`
* OS fork (New thread created with a clone of thread state):
* - clone{2, 3}
* - Thread = CreateThread(0, 0, CopyOfThreadState, PPID);
* - ExecutionThread(Thread); // Starts executing without creating another host thread
* Thunk callback executing guest code from native host thread
* - Thread = CreateThread(0, 0, NewState, PPID);
* - InitializeThreadTLSData(Thread);
* - HandleCallback(Thread, RIP);
*/
void HandleCallback(FEXCore::Core::InternalThreadState *Thread, uint64_t RIP) override;
FEXCore::Core::InternalThreadState*
CreateThread(uint64_t InitialRIP, uint64_t StackPointer, FEXCore::Core::CPUState* NewThreadState, uint64_t ParentTID) override;
uint64_t RestoreRIPFromHostPC(FEXCore::Core::InternalThreadState *Thread, uint64_t HostPC) override;
uint32_t ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadState *Thread) override;
void SetFlagsFromCompactedEFLAGS(FEXCore::Core::InternalThreadState *Thread, uint32_t EFLAGS) override;
// Public for threading
void ExecutionThread(FEXCore::Core::InternalThreadState* Thread) override;
/**
* @brief Used to create FEX thread objects in preparation for creating a true OS thread. Does set a TID or PID.
*
* @param NewThreadState The initial thread state to setup for our state
* @param ParentTID The PID that was the parent thread that created this
*
* @return The InternalThreadState object that tracks all of the emulated thread's state
*
* Usecases:
* OS thread Creation:
* - Thread = CreateThread(NewState, PPID);
* - InitializeThread(Thread);
* OS fork (New thread created with a clone of thread state):
* - clone{2, 3}
* - Thread = CreateThread(CopyOfThreadState, PPID);
* - ExecutionThread(Thread); // Starts executing without creating another host thread
* Thunk callback executing guest code from native host thread
* - Thread = CreateThread(NewState, PPID);
* - InitializeThreadTLSData(Thread);
* - HandleCallback(Thread, RIP);
*/
FEXCore::Core::InternalThreadState* CreateThread(FEXCore::Core::CPUState *NewThreadState, uint64_t ParentTID) override;
// Public for threading
void ExecutionThread(FEXCore::Core::InternalThreadState *Thread) override;
/**
* @brief Initializes the OS thread object and prepares to start executing on that new OS thread
*
* @param Thread The internal FEX thread state object
*
* The OS thread will wait until RunThread is executed
*/
void InitializeThread(FEXCore::Core::InternalThreadState *Thread) override;
/**
* @brief Starts the OS thread object to start executing guest code
*
* @param Thread The internal FEX thread state object
*/
void RunThread(FEXCore::Core::InternalThreadState *Thread) override;
void StopThread(FEXCore::Core::InternalThreadState *Thread) override;
/**
* @brief Destroys this FEX thread object and stops tracking it internally
*
* @param Thread The internal FEX thread state object
*/
void DestroyThread(FEXCore::Core::InternalThreadState *Thread) override;
/**
* @brief Destroys this FEX thread object and stops tracking it internally
*
* @param Thread The internal FEX thread state object
*/
void DestroyThread(FEXCore::Core::InternalThreadState* Thread, bool NeedsTLSUninstall) override;
#ifndef _WIN32
void LockBeforeFork(FEXCore::Core::InternalThreadState *Thread) override;
void UnlockAfterFork(FEXCore::Core::InternalThreadState *Thread, bool Child) override;
void LockBeforeFork(FEXCore::Core::InternalThreadState* Thread) override;
void UnlockAfterFork(FEXCore::Core::InternalThreadState* Thread, bool Child) override;
#endif
void SetSignalDelegator(FEXCore::SignalDelegator *SignalDelegation) override;
void SetSyscallHandler(FEXCore::HLE::SyscallHandler *Handler) override;
void SetSignalDelegator(FEXCore::SignalDelegator* SignalDelegation) override;
void SetSyscallHandler(FEXCore::HLE::SyscallHandler* Handler) override;
FEXCore::CPUID::FunctionResults RunCPUIDFunction(uint32_t Function, uint32_t Leaf) override;
FEXCore::CPUID::XCRResults RunXCRFunction(uint32_t Function) override;
FEXCore::CPUID::FunctionResults RunCPUIDFunctionName(uint32_t Function, uint32_t Leaf, uint32_t CPU) override;
FEXCore::CPUID::FunctionResults RunCPUIDFunction(uint32_t Function, uint32_t Leaf) override;
FEXCore::CPUID::XCRResults RunXCRFunction(uint32_t Function) override;
FEXCore::CPUID::FunctionResults RunCPUIDFunctionName(uint32_t Function, uint32_t Leaf, uint32_t CPU) override;
FEXCore::IR::AOTIRCacheEntry *LoadAOTIRCacheEntry(const fextl::string& Name) override;
void UnloadAOTIRCacheEntry(FEXCore::IR::AOTIRCacheEntry *Entry) override;
FEXCore::IR::AOTIRCacheEntry* LoadAOTIRCacheEntry(const fextl::string& Name) override;
void UnloadAOTIRCacheEntry(FEXCore::IR::AOTIRCacheEntry* Entry) override;
void SetAOTIRLoader(AOTIRLoaderCBFn CacheReader) override {
IRCaptureCache.SetAOTIRLoader(std::move(CacheReader));
}
void SetAOTIRWriter(AOTIRWriterCBFn CacheWriter) override {
IRCaptureCache.SetAOTIRWriter(std::move(CacheWriter));
}
void SetAOTIRRenamer(AOTIRRenamerCBFn CacheRenamer) override {
IRCaptureCache.SetAOTIRRenamer(std::move(CacheRenamer));
}
void SetAOTIRLoader(AOTIRLoaderCBFn CacheReader) override {
IRCaptureCache.SetAOTIRLoader(std::move(CacheReader));
}
void SetAOTIRWriter(AOTIRWriterCBFn CacheWriter) override {
IRCaptureCache.SetAOTIRWriter(std::move(CacheWriter));
}
void SetAOTIRRenamer(AOTIRRenamerCBFn CacheRenamer) override {
IRCaptureCache.SetAOTIRRenamer(std::move(CacheRenamer));
}
void FinalizeAOTIRCache() override {
IRCaptureCache.FinalizeAOTIRCache();
}
void WriteFilesWithCode(AOTIRCodeFileWriterFn Writer) override {
IRCaptureCache.WriteFilesWithCode(Writer);
}
void InvalidateGuestCodeRange(FEXCore::Core::InternalThreadState *Thread, uint64_t Start, uint64_t Length) override;
void InvalidateGuestCodeRange(FEXCore::Core::InternalThreadState *Thread, uint64_t Start, uint64_t Length, CodeRangeInvalidationFn callback) override;
void MarkMemoryShared() override;
void FinalizeAOTIRCache() override {
IRCaptureCache.FinalizeAOTIRCache();
}
void WriteFilesWithCode(AOTIRCodeFileWriterFn Writer) override {
IRCaptureCache.WriteFilesWithCode(Writer);
}
void ConfigureAOTGen(FEXCore::Core::InternalThreadState *Thread, fextl::set<uint64_t> *ExternalBranches, uint64_t SectionMaxAddress) override;
// returns false if a handler was already registered
CustomIRResult AddCustomIREntrypoint(uintptr_t Entrypoint, CustomIREntrypointHandler Handler, void *Creator = nullptr, void *Data = nullptr) override;
void ClearCodeCache(FEXCore::Core::InternalThreadState* Thread) override;
void InvalidateGuestCodeRange(FEXCore::Core::InternalThreadState* Thread, uint64_t Start, uint64_t Length) override;
void InvalidateGuestCodeRange(FEXCore::Core::InternalThreadState* Thread, uint64_t Start, uint64_t Length, CodeRangeInvalidationFn callback) override;
FEXCore::ForkableSharedMutex& GetCodeInvalidationMutex() override {
return CodeInvalidationMutex;
}
void AppendThunkDefinitions(fextl::vector<FEXCore::IR::ThunkDefinition> const& Definitions) override;
void MarkMemoryShared(FEXCore::Core::InternalThreadState* Thread) override;
public:
friend class FEXCore::HLE::SyscallHandler;
#ifdef JIT_ARM64
friend class FEXCore::CPU::Arm64JITCore;
#endif
#ifdef JIT_X86_64
friend class FEXCore::CPU::X86JITCore;
#endif
void ConfigureAOTGen(FEXCore::Core::InternalThreadState* Thread, fextl::set<uint64_t>* ExternalBranches, uint64_t SectionMaxAddress) override;
friend class FEXCore::IR::Validation::IRValidation;
bool IsAddressInCodeBuffer(FEXCore::Core::InternalThreadState* Thread, uintptr_t Address) const override;
struct {
CoreRunningMode RunningMode {CoreRunningMode::MODE_RUN};
uint64_t VirtualMemSize{1ULL << 36};
// returns false if a handler was already registered
CustomIRResult AddCustomIREntrypoint(uintptr_t Entrypoint, CustomIREntrypointHandler Handler, void* Creator = nullptr, void* Data = nullptr);
// this is for internal use
bool ValidateIRarser { false };
void AppendThunkDefinitions(const fextl::vector<FEXCore::IR::ThunkDefinition>& Definitions) override;
FEX_CONFIG_OPT(Multiblock, MULTIBLOCK);
FEX_CONFIG_OPT(SingleStepConfig, SINGLESTEP);
FEX_CONFIG_OPT(GdbServer, GDBSERVER);
FEX_CONFIG_OPT(Is64BitMode, IS64BIT_MODE);
FEX_CONFIG_OPT(TSOEnabled, TSOENABLED);
FEX_CONFIG_OPT(TSOAutoMigration, TSOAUTOMIGRATION);
FEX_CONFIG_OPT(ABILocalFlags, ABILOCALFLAGS);
FEX_CONFIG_OPT(AOTIRCapture, AOTIRCAPTURE);
FEX_CONFIG_OPT(AOTIRGenerate, AOTIRGENERATE);
FEX_CONFIG_OPT(AOTIRLoad, AOTIRLOAD);
FEX_CONFIG_OPT(SMCChecks, SMCCHECKS);
FEX_CONFIG_OPT(Core, CORE);
FEX_CONFIG_OPT(MaxInstPerBlock, MAXINST);
FEX_CONFIG_OPT(RootFSPath, ROOTFS);
FEX_CONFIG_OPT(ThunkHostLibsPath, THUNKHOSTLIBS);
FEX_CONFIG_OPT(ThunkHostLibsPath32, THUNKHOSTLIBS32);
FEX_CONFIG_OPT(ThunkConfigFile, THUNKCONFIG);
FEX_CONFIG_OPT(StaticRegisterAllocation, SRA);
FEX_CONFIG_OPT(GlobalJITNaming, GLOBALJITNAMING);
FEX_CONFIG_OPT(LibraryJITNaming, LIBRARYJITNAMING);
FEX_CONFIG_OPT(BlockJITNaming, BLOCKJITNAMING);
FEX_CONFIG_OPT(GDBSymbols, GDBSYMBOLS);
FEX_CONFIG_OPT(ParanoidTSO, PARANOIDTSO);
FEX_CONFIG_OPT(CacheObjectCodeCompilation, CACHEOBJECTCODECOMPILATION);
FEX_CONFIG_OPT(x87ReducedPrecision, X87REDUCEDPRECISION);
FEX_CONFIG_OPT(DisableTelemetry, DISABLETELEMETRY);
FEX_CONFIG_OPT(DisableVixlIndirectCalls, DISABLE_VIXL_INDIRECT_RUNTIME_CALLS);
} Config;
public:
friend class FEXCore::HLE::SyscallHandler;
#ifdef JIT_ARM64
friend class FEXCore::CPU::Arm64JITCore;
#endif
FEXCore::HostFeatures HostFeatures;
friend class FEXCore::IR::Validation::IRValidation;
std::mutex ThreadCreationMutex;
FEXCore::Core::InternalThreadState* ParentThread{};
fextl::vector<FEXCore::Core::InternalThreadState*> Threads;
std::atomic_bool CoreShuttingDown{false};
bool NeedToCheckXID{true};
struct {
CoreRunningMode RunningMode {CoreRunningMode::MODE_RUN};
uint64_t VirtualMemSize {1ULL << 36};
std::mutex IdleWaitMutex;
std::condition_variable IdleWaitCV;
std::atomic<uint32_t> IdleWaitRefCount{};
// Used if the JIT needs to have its interrupt fault code emitted.
bool NeedsPendingInterruptFaultCheck {false};
Event PauseWait;
bool Running{};
FEX_CONFIG_OPT(Multiblock, MULTIBLOCK);
FEX_CONFIG_OPT(SingleStepConfig, SINGLESTEP);
FEX_CONFIG_OPT(GdbServer, GDBSERVER);
FEX_CONFIG_OPT(Is64BitMode, IS64BIT_MODE);
FEX_CONFIG_OPT(TSOEnabled, TSOENABLED);
FEX_CONFIG_OPT(TSOAutoMigration, TSOAUTOMIGRATION);
FEX_CONFIG_OPT(ABILocalFlags, ABILOCALFLAGS);
FEX_CONFIG_OPT(AOTIRCapture, AOTIRCAPTURE);
FEX_CONFIG_OPT(AOTIRGenerate, AOTIRGENERATE);
FEX_CONFIG_OPT(AOTIRLoad, AOTIRLOAD);
FEX_CONFIG_OPT(SMCChecks, SMCCHECKS);
FEX_CONFIG_OPT(Core, CORE);
FEX_CONFIG_OPT(MaxInstPerBlock, MAXINST);
FEX_CONFIG_OPT(RootFSPath, ROOTFS);
FEX_CONFIG_OPT(ThunkHostLibsPath, THUNKHOSTLIBS);
FEX_CONFIG_OPT(ThunkHostLibsPath32, THUNKHOSTLIBS32);
FEX_CONFIG_OPT(ThunkConfigFile, THUNKCONFIG);
FEX_CONFIG_OPT(GlobalJITNaming, GLOBALJITNAMING);
FEX_CONFIG_OPT(LibraryJITNaming, LIBRARYJITNAMING);
FEX_CONFIG_OPT(BlockJITNaming, BLOCKJITNAMING);
FEX_CONFIG_OPT(GDBSymbols, GDBSYMBOLS);
FEX_CONFIG_OPT(ParanoidTSO, PARANOIDTSO);
FEX_CONFIG_OPT(CacheObjectCodeCompilation, CACHEOBJECTCODECOMPILATION);
FEX_CONFIG_OPT(x87ReducedPrecision, X87REDUCEDPRECISION);
FEX_CONFIG_OPT(DisableTelemetry, DISABLETELEMETRY);
FEX_CONFIG_OPT(DisableVixlIndirectCalls, DISABLE_VIXL_INDIRECT_RUNTIME_CALLS);
FEX_CONFIG_OPT(SmallTSCScale, SMALLTSCSCALE);
} Config;
FEXCore::ForkableSharedMutex CodeInvalidationMutex;
FEXCore::CPUIDEmu CPUID;
FEXCore::HLE::SyscallHandler *SyscallHandler{};
FEXCore::HLE::SourcecodeResolver *SourcecodeResolver{};
fextl::unique_ptr<FEXCore::ThunkHandler> ThunkHandler;
fextl::unique_ptr<FEXCore::CPU::Dispatcher> Dispatcher;
std::atomic_bool CoreShuttingDown {false};
CustomCPUFactoryType CustomCPUFactory;
FEXCore::Context::ExitHandler CustomExitHandler;
FEXCore::ForkableSharedMutex CodeInvalidationMutex;
FEXCore::HostFeatures HostFeatures;
// CPUID depends on HostFeatures so needs to be initialized after that.
FEXCore::CPUIDEmu CPUID;
FEXCore::HLE::SyscallHandler* SyscallHandler {};
FEXCore::HLE::SourcecodeResolver* SourcecodeResolver {};
fextl::unique_ptr<FEXCore::ThunkHandler> ThunkHandler;
fextl::unique_ptr<FEXCore::CPU::Dispatcher> Dispatcher;
CustomCPUFactoryType CustomCPUFactory;
FEXCore::Context::ExitHandler CustomExitHandler;
#ifdef BLOCKSTATS
fextl::unique_ptr<FEXCore::BlockSamplingData> BlockData;
fextl::unique_ptr<FEXCore::BlockSamplingData> BlockData;
#endif
SignalDelegator *SignalDelegation{};
X86GeneratedCode X86CodeGen;
SignalDelegator* SignalDelegation {};
X86GeneratedCode X86CodeGen;
ContextImpl();
~ContextImpl();
ContextImpl();
~ContextImpl();
bool IsPaused() const { return !Running; }
void WaitForThreadsToRun();
void Stop(bool IgnoreCurrentThread);
void WaitForIdle();
void SignalThread(FEXCore::Core::InternalThreadState *Thread, FEXCore::Core::SignalEvent Event);
static void ThreadRemoveCodeEntry(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP);
static void ThreadAddBlockLink(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestDestination,
FEXCore::Context::ExitFunctionLinkData* HostLink, const BlockDelinkerFunc& delinker);
bool GetGdbServerStatus() const { return DebugServer != nullptr; }
void StartGdbServer();
void StopGdbServer();
template<auto Fn>
static uint64_t ThreadExitFunctionLink(FEXCore::Core::CpuStateFrame* Frame, ExitFunctionLinkData* Record) {
auto Thread = Frame->Thread;
auto lk = GuardSignalDeferringSection<std::shared_lock>(static_cast<ContextImpl*>(Thread->CTX)->CodeInvalidationMutex, Thread);
static void ThreadRemoveCodeEntry(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP);
static void ThreadAddBlockLink(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestDestination, uintptr_t HostLink, const std::function<void()> &delinker);
return Fn(Frame, Record);
}
template<auto Fn>
static uint64_t ThreadExitFunctionLink(FEXCore::Core::CpuStateFrame *Frame, uint64_t *record) {
auto Thread = Frame->Thread;
ScopedDeferredSignalWithForkableSharedLock lk(static_cast<ContextImpl*>(Thread->CTX)->CodeInvalidationMutex, Thread);
// Wrapper which takes CpuStateFrame instead of InternalThreadState and unique_locks CodeInvalidationMutex
// Must be called from owning thread
static void ThreadRemoveCodeEntryFromJit(FEXCore::Core::CpuStateFrame* Frame, uint64_t GuestRIP) {
auto Thread = Frame->Thread;
return Fn(Frame, record);
}
LOGMAN_THROW_A_FMT(Thread->ThreadManager.GetTID() == FHU::Syscalls::gettid(), "Must be called from owning thread {}, not {}",
Thread->ThreadManager.GetTID(), FHU::Syscalls::gettid());
auto lk = GuardSignalDeferringSection(static_cast<ContextImpl*>(Thread->CTX)->CodeInvalidationMutex, Thread);
// Wrapper which takes CpuStateFrame instead of InternalThreadState and unique_locks CodeInvalidationMutex
// Must be called from owning thread
static void ThreadRemoveCodeEntryFromJit(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP) {
auto Thread = Frame->Thread;
ThreadRemoveCodeEntry(Thread, GuestRIP);
}
LogMan::Throw::AFmt(Thread->ThreadManager.GetTID() == FHU::Syscalls::gettid(), "Must be called from owning thread {}, not {}", Thread->ThreadManager.GetTID(), FHU::Syscalls::gettid());
ScopedDeferredSignalWithForkableUniqueLock lk(static_cast<ContextImpl*>(Thread->CTX)->CodeInvalidationMutex, Thread);
void RemoveCustomIREntrypoint(uintptr_t Entrypoint);
ThreadRemoveCodeEntry(Thread, GuestRIP);
}
void RemoveCustomIREntrypoint(uintptr_t Entrypoint);
struct GenerateIRResult {
FEXCore::IR::IRListView* IRList;
FEXCore::IR::RegisterAllocationData::UniquePtr RAData;
uint64_t TotalInstructions;
uint64_t TotalInstructionsLength;
uint64_t StartAddr;
uint64_t Length;
};
[[nodiscard]] GenerateIRResult GenerateIR(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP, bool ExtendedDebugInfo);
struct CompileCodeResult {
void* CompiledCode;
FEXCore::IR::IRListView* IRData;
FEXCore::Core::DebugData* DebugData;
FEXCore::IR::RegisterAllocationData::UniquePtr RAData;
bool GeneratedIR;
uint64_t StartAddr;
uint64_t Length;
};
[[nodiscard]] CompileCodeResult CompileCode(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP);
uintptr_t CompileBlock(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP);
// same as CompileBlock, but aborts on failure
void CompileBlockJit(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP);
// Used for thread creation from syscalls
/**
* @brief Initializes TID, PID and TLS data for a thread
*
* @param Thread The internal FEX thread state object
*/
void InitializeThreadTLSData(FEXCore::Core::InternalThreadState *Thread);
void CopyMemoryMapping(FEXCore::Core::InternalThreadState *ParentThread, FEXCore::Core::InternalThreadState *ChildThread);
fextl::vector<FEXCore::Core::InternalThreadState*>* GetThreads() { return &Threads; }
uint8_t GetGPRSize() const { return Config.Is64BitMode ? 8 : 4; }
FEXCore::JITSymbols Symbols;
void GetVDSOSigReturn(VDSOSigReturn *VDSOPointers) override {
if (VDSOPointers->VDSO_kernel_sigreturn == nullptr) {
VDSOPointers->VDSO_kernel_sigreturn = reinterpret_cast<void*>(X86CodeGen.sigreturn_32);
}
if (VDSOPointers->VDSO_kernel_rt_sigreturn == nullptr) {
VDSOPointers->VDSO_kernel_rt_sigreturn = reinterpret_cast<void*>(X86CodeGen.rt_sigreturn_32);
}
}
void IncrementIdleRefCount() override {
++IdleWaitRefCount;
}
FEXCore::Utils::PooledAllocatorVirtual OpDispatcherAllocator;
FEXCore::Utils::PooledAllocatorVirtual FrontendAllocator;
// If Atomic-based TSO emulation is enabled or not.
bool IsAtomicTSOEnabled() const { return AtomicTSOEmulationEnabled; }
void SetHardwareTSOSupport(bool HardwareTSOSupported) override {
SupportsHardwareTSO = HardwareTSOSupported;
UpdateAtomicTSOEmulationConfig();
}
// Returns if Software TSO emulation is required.
// NOTE: This doesn't necessary return if Atomic-based TSO is currently enabled.
// This will still return true if on a single thread and TSO is currently disabled.
//
// This is to ensure that if early initialization checks CPU features and TSO /could/ be enabled, that
// we return consistent results.
//
// To check if Atomic TSO is currently enabled in the JIT, use `IsAtomicTSOEnabled` instead.
bool SoftwareTSORequired() const {
if (SupportsHardwareTSO) return false;
return Config.TSOEnabled;
}
void EnableExitOnHLT() override { ExitOnHLT = true; }
bool ExitOnHLTEnabled() const { return ExitOnHLT; }
FEXCore::CPU::CPUBackendFeatures BackendFeatures;
protected:
void ClearCodeCache(FEXCore::Core::InternalThreadState *Thread);
void UpdateAtomicTSOEmulationConfig() {
if (SupportsHardwareTSO) {
// If the hardware supports TSO then we don't need to emulate it through atomics.
AtomicTSOEmulationEnabled = false;
}
else {
// Atomic TSO emulation only enabled if the config option is enabled.
AtomicTSOEmulationEnabled = (IsMemoryShared || !Config.TSOAutoMigration) && Config.TSOEnabled;
}
}
private:
/**
* @brief Does some final thread initialization
*
* @param Thread The internal FEX thread state object
*
* InitCore and CreateThread both call this to finish up thread object initialization
*/
void InitializeThreadData(FEXCore::Core::InternalThreadState *Thread);
/**
* @brief Initializes the JIT compilers for the thread
*
* @param State The internal FEX thread state object
*
* InitializeCompiler is called inside of CreateThread, so you likely don't need this
*/
void InitializeCompiler(FEXCore::Core::InternalThreadState* Thread);
void WaitForIdleWithTimeout();
void NotifyPause();
void AddBlockMapping(FEXCore::Core::InternalThreadState *Thread, uint64_t Address, void *Ptr);
// Entry Cache
std::mutex ExitMutex;
fextl::unique_ptr<GdbServer> DebugServer;
IR::AOTIRCaptureCache IRCaptureCache;
fextl::unique_ptr<FEXCore::CodeSerialize::CodeObjectSerializeService> CodeObjectCacheService;
bool StartPaused = false;
bool IsMemoryShared = false;
bool SupportsHardwareTSO = false;
bool AtomicTSOEmulationEnabled = true;
bool ExitOnHLT = false;
FEX_CONFIG_OPT(AppFilename, APP_FILENAME);
std::shared_mutex CustomIRMutex;
fextl::unordered_map<uint64_t, std::tuple<CustomIREntrypointHandler, void *, void *>> CustomIRHandlers;
FEXCore::CPU::DispatcherConfig DispatcherConfig;
struct GenerateIRResult {
fextl::unique_ptr<FEXCore::IR::IRStorageBase> IR;
uint64_t TotalInstructions;
uint64_t TotalInstructionsLength;
uint64_t StartAddr;
uint64_t Length;
};
[[nodiscard]]
GenerateIRResult GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP, bool ExtendedDebugInfo, uint64_t MaxInst);
uint64_t HandleSyscall(FEXCore::HLE::SyscallHandler *Handler, FEXCore::Core::CpuStateFrame *Frame, FEXCore::HLE::SyscallArguments *Args);
}
struct CompileCodeResult {
void* CompiledCode;
fextl::unique_ptr<FEXCore::IR::IRStorageBase> IR;
FEXCore::Core::DebugData* DebugData;
bool GeneratedIR;
uint64_t StartAddr;
uint64_t Length;
};
[[nodiscard]]
CompileCodeResult CompileCode(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP, uint64_t MaxInst = 0);
uintptr_t CompileBlock(FEXCore::Core::CpuStateFrame* Frame, uint64_t GuestRIP, uint64_t MaxInst = 0);
// Used for thread creation from syscalls
/**
* @brief Initializes TID, PID and TLS data for a thread
*
* @param Thread The internal FEX thread state object
*/
void InitializeThreadTLSData(FEXCore::Core::InternalThreadState* Thread);
void CopyMemoryMapping(FEXCore::Core::InternalThreadState* ParentThread, FEXCore::Core::InternalThreadState* ChildThread);
uint8_t GetGPRSize() const {
return Config.Is64BitMode ? 8 : 4;
}
FEXCore::JITSymbols Symbols;
void GetVDSOSigReturn(VDSOSigReturn* VDSOPointers) override {
if (VDSOPointers->VDSO_kernel_sigreturn == nullptr) {
VDSOPointers->VDSO_kernel_sigreturn = reinterpret_cast<void*>(X86CodeGen.sigreturn_32);
}
if (VDSOPointers->VDSO_kernel_rt_sigreturn == nullptr) {
VDSOPointers->VDSO_kernel_rt_sigreturn = reinterpret_cast<void*>(X86CodeGen.rt_sigreturn_32);
}
}
FEXCore::Utils::PooledAllocatorVirtual OpDispatcherAllocator;
FEXCore::Utils::PooledAllocatorVirtual FrontendAllocator;
// If Atomic-based TSO emulation is enabled or not.
bool IsAtomicTSOEnabled() const {
return AtomicTSOEmulationEnabled;
}
void SetHardwareTSOSupport(bool HardwareTSOSupported) override {
SupportsHardwareTSO = HardwareTSOSupported;
UpdateAtomicTSOEmulationConfig();
}
// Returns if Software TSO emulation is required.
// NOTE: This doesn't necessary return if Atomic-based TSO is currently enabled.
// This will still return true if on a single thread and TSO is currently disabled.
//
// This is to ensure that if early initialization checks CPU features and TSO /could/ be enabled, that
// we return consistent results.
//
// To check if Atomic TSO is currently enabled in the JIT, use `IsAtomicTSOEnabled` instead.
bool SoftwareTSORequired() const {
if (SupportsHardwareTSO) {
return false;
}
return Config.TSOEnabled;
}
void EnableExitOnHLT() override {
ExitOnHLT = true;
}
bool ExitOnHLTEnabled() const {
return ExitOnHLT;
}
FEXCore::CPU::CPUBackendFeatures BackendFeatures;
protected:
void UpdateAtomicTSOEmulationConfig() {
if (SupportsHardwareTSO) {
// If the hardware supports TSO then we don't need to emulate it through atomics.
AtomicTSOEmulationEnabled = false;
} else {
// Atomic TSO emulation only enabled if the config option is enabled.
AtomicTSOEmulationEnabled = (IsMemoryShared || !Config.TSOAutoMigration) && Config.TSOEnabled;
}
}
private:
/**
* @brief Initializes the JIT compilers for the thread
*
* @param State The internal FEX thread state object
*
* InitializeCompiler is called inside of CreateThread, so you likely don't need this
*/
void InitializeCompiler(FEXCore::Core::InternalThreadState* Thread);
void AddBlockMapping(FEXCore::Core::InternalThreadState* Thread, uint64_t Address, void* Ptr);
IR::AOTIRCaptureCache IRCaptureCache;
fextl::unique_ptr<FEXCore::CodeSerialize::CodeObjectSerializeService> CodeObjectCacheService;
bool StartPaused = false;
bool IsMemoryShared = false;
bool SupportsHardwareTSO = false;
bool AtomicTSOEmulationEnabled = true;
bool ExitOnHLT = false;
FEX_CONFIG_OPT(AppFilename, APP_FILENAME);
std::shared_mutex CustomIRMutex;
std::atomic<bool> HasCustomIRHandlers {};
fextl::unordered_map<uint64_t, std::tuple<CustomIREntrypointHandler, void*, void*>> CustomIRHandlers;
};
} // namespace FEXCore::Context
File diff suppressed because it is too large. Load diff
@@ -2,9 +2,6 @@
#pragma once
#include "FEXCore/Utils/EnumUtils.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Registers.h"
#include "Interface/Core/ObjectCache/Relocations.h"
#include <aarch64/assembler-aarch64.h>
@@ -21,6 +18,9 @@
#endif
#include <FEXCore/Config/Config.h>
#include <FEXCore/fextl/vector.h>
#include <CodeEmitter/Emitter.h>
#include <CodeEmitter/Registers.h>
#include <array>
#include <cstddef>
@@ -34,64 +34,74 @@ class ContextImpl;
namespace FEXCore::CPU {
// Contains the address to the currently available CPU state
constexpr auto STATE = FEXCore::ARMEmitter::XReg::x28;
constexpr auto STATE = ARMEmitter::XReg::x28;
#ifndef _M_ARM_64EC
// GPR temporaries. Only x3 can be used across spill boundaries
// so if these ever need to change, be very careful about that.
constexpr auto TMP1 = FEXCore::ARMEmitter::XReg::x0;
constexpr auto TMP2 = FEXCore::ARMEmitter::XReg::x1;
constexpr auto TMP3 = FEXCore::ARMEmitter::XReg::x2;
constexpr auto TMP4 = FEXCore::ARMEmitter::XReg::x3;
constexpr auto TMP1 = ARMEmitter::XReg::x0;
constexpr auto TMP2 = ARMEmitter::XReg::x1;
constexpr auto TMP3 = ARMEmitter::XReg::x2;
constexpr auto TMP4 = ARMEmitter::XReg::x3;
constexpr bool TMP_ABIARGS = true;
// We pin r26/r27 as PF/AF respectively, this is internal FEX ABI.
constexpr auto REG_PF = ARMEmitter::Reg::r26;
constexpr auto REG_AF = ARMEmitter::Reg::r27;
// Vector temporaries
constexpr auto VTMP1 = FEXCore::ARMEmitter::VReg::v0;
constexpr auto VTMP2 = FEXCore::ARMEmitter::VReg::v1;
constexpr auto VTMP1 = ARMEmitter::VReg::v0;
constexpr auto VTMP2 = ARMEmitter::VReg::v1;
#else
constexpr auto TMP1 = ARMEmitter::XReg::x10;
constexpr auto TMP2 = ARMEmitter::XReg::x11;
constexpr auto TMP3 = ARMEmitter::XReg::x12;
constexpr auto TMP4 = ARMEmitter::XReg::x13;
constexpr bool TMP_ABIARGS = false;
// We pin r11/r12 as PF/AF respectively for arm64ec, as r26/r27 are used for SRA.
constexpr auto REG_PF = ARMEmitter::Reg::r9;
constexpr auto REG_AF = ARMEmitter::Reg::r24;
// Vector temporaries
constexpr auto VTMP1 = ARMEmitter::VReg::v16;
constexpr auto VTMP2 = ARMEmitter::VReg::v17;
// Entry/Exit ABI
constexpr auto EC_CALL_CHECKER_PC_REG = ARMEmitter::XReg::x9;
constexpr auto EC_ENTRY_CPUAREA_REG = ARMEmitter::XReg::x17;
#endif
// Predicate register temporaries (used when AVX support is enabled)
// PRED_TMP_16B indicates a predicate register that indicates the first 16 bytes set to 1.
// PRED_TMP_32B indicates a predicate register that indicates the first 32 bytes set to 1.
constexpr FEXCore::ARMEmitter::PRegister PRED_TMP_16B = FEXCore::ARMEmitter::PReg::p6;
constexpr FEXCore::ARMEmitter::PRegister PRED_TMP_32B = FEXCore::ARMEmitter::PReg::p7;
constexpr ARMEmitter::PRegister PRED_TMP_16B = ARMEmitter::PReg::p6;
constexpr ARMEmitter::PRegister PRED_TMP_32B = ARMEmitter::PReg::p7;
// This class contains common emitter utility functions that can
// be used by both Arm64 JIT and ARM64 Dispatcher
class Arm64Emitter : public FEXCore::ARMEmitter::Emitter {
class Arm64Emitter : public ARMEmitter::Emitter {
protected:
Arm64Emitter(FEXCore::Context::ContextImpl *ctx, size_t size);
~Arm64Emitter();
Arm64Emitter(FEXCore::Context::ContextImpl* ctx, void* EmissionPtr = nullptr, size_t size = 0);
FEXCore::Context::ContextImpl *EmitterCTX;
FEXCore::Context::ContextImpl* EmitterCTX;
vixl::aarch64::CPU CPU;
std::span<const FEXCore::ARMEmitter::Register> ConfiguredDynamicRegisterBase{};
std::span<const FEXCore::ARMEmitter::Register> StaticRegisters{};
std::span<const FEXCore::ARMEmitter::Register> GeneralRegisters{};
std::span<const std::pair<FEXCore::ARMEmitter::Register, FEXCore::ARMEmitter::Register>> GeneralPairRegisters{};
std::span<const FEXCore::ARMEmitter::VRegister> StaticFPRegisters{};
std::span<const FEXCore::ARMEmitter::VRegister> GeneralFPRegisters{};
std::span<const ARMEmitter::Register> ConfiguredDynamicRegisterBase {};
std::span<const ARMEmitter::Register> StaticRegisters {};
std::span<const ARMEmitter::Register> GeneralRegisters {};
std::span<const ARMEmitter::VRegister> StaticFPRegisters {};
std::span<const ARMEmitter::VRegister> GeneralFPRegisters {};
uint32_t PairRegisters = 0;
/**
* @name Register Allocation
* @{ */
constexpr static uint32_t RegisterClasses = 6;
constexpr static uint64_t GPRBase = (0ULL << 32);
constexpr static uint64_t FPRBase = (1ULL << 32);
constexpr static uint64_t GPRPairBase = (2ULL << 32);
/** @} */
constexpr static uint8_t RA_32 = 0;
constexpr static uint8_t RA_64 = 1;
constexpr static uint8_t RA_FPR = 2;
void LoadConstant(FEXCore::ARMEmitter::Size s, FEXCore::ARMEmitter::Register Reg, uint64_t Constant, bool NOPPad = false);
void LoadConstant(ARMEmitter::Size s, ARMEmitter::Register Reg, uint64_t Constant, bool NOPPad = false);
// NOTE: These functions WILL clobber the register TMP4 if AVX support is enabled
// and FPRs are being spilled or filled. If only GPRs are spilled/filled, then
// TMP4 is left alone.
void SpillStaticRegs(FEXCore::ARMEmitter::Register TmpReg, bool FPRs = true, uint32_t GPRSpillMask = ~0U, uint32_t FPRSpillMask = ~0U);
void SpillStaticRegs(ARMEmitter::Register TmpReg, bool FPRs = true, uint32_t GPRSpillMask = ~0U, uint32_t FPRSpillMask = ~0U);
void FillStaticRegs(bool FPRs = true, uint32_t GPRFillMask = ~0U, uint32_t FPRFillMask = ~0U);
// Register 0-18 + 29 + 30 are caller saved
@@ -102,13 +112,13 @@ protected:
static constexpr uint32_t CALLER_FPR_MASK = ~0U;
// Generic push and pop vector registers.
void PushVectorRegisters(FEXCore::ARMEmitter::Register TmpReg, bool SVERegs, std::span<const FEXCore::ARMEmitter::VRegister> VRegs);
void PushGeneralRegisters(FEXCore::ARMEmitter::Register TmpReg, std::span<const FEXCore::ARMEmitter::Register> Regs);
void PushVectorRegisters(ARMEmitter::Register TmpReg, bool SVERegs, std::span<const ARMEmitter::VRegister> VRegs);
void PushGeneralRegisters(ARMEmitter::Register TmpReg, std::span<const ARMEmitter::Register> Regs);
void PopVectorRegisters(bool SVERegs, std::span<const FEXCore::ARMEmitter::VRegister> VRegs);
void PopGeneralRegisters(std::span<const FEXCore::ARMEmitter::Register> Regs);
void PopVectorRegisters(bool SVERegs, std::span<const ARMEmitter::VRegister> VRegs);
void PopGeneralRegisters(std::span<const ARMEmitter::Register> Regs);
void PushDynamicRegsAndLR(FEXCore::ARMEmitter::Register TmpReg);
void PushDynamicRegsAndLR(ARMEmitter::Register TmpReg);
void PopDynamicRegsAndLR();
void PushCalleeSavedRegisters();
@@ -124,26 +134,24 @@ protected:
// Callee Saved:
// - X9-X15, X19-X31
// - Low 128-bits of v8-v31
void SpillForPreserveAllABICall(FEXCore::ARMEmitter::Register TmpReg, bool FPRs = true);
void SpillForPreserveAllABICall(ARMEmitter::Register TmpReg, bool FPRs = true);
void FillForPreserveAllABICall(bool FPRs = true);
void SpillForABICall(bool SupportsPreserveAllABI, FEXCore::ARMEmitter::Register TmpReg, bool FPRs = true) {
void SpillForABICall(bool SupportsPreserveAllABI, ARMEmitter::Register TmpReg, bool FPRs = true) {
if (SupportsPreserveAllABI) {
SpillForPreserveAllABICall(TMP1, true);
}
else {
SpillStaticRegs(TMP1);
PushDynamicRegsAndLR(TMP1);
SpillForPreserveAllABICall(TmpReg, FPRs);
} else {
SpillStaticRegs(TmpReg, FPRs);
PushDynamicRegsAndLR(TmpReg);
}
}
void FillForABICall(bool SupportsPreserveAllABI, bool FPRs = true) {
if (SupportsPreserveAllABI) {
FillForPreserveAllABICall(true);
}
else {
FillForPreserveAllABICall(FPRs);
} else {
PopDynamicRegsAndLR();
FillStaticRegs();
FillStaticRegs(FPRs);
}
}
@@ -163,8 +171,7 @@ protected:
template<typename R, typename... P>
void GenerateRuntimeCall(R (*Function)(P...)) {
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(
&(vixl::aarch64::Simulator::RuntimeCallStructHelper<R, P...>::Wrapper));
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(&(vixl::aarch64::Simulator::RuntimeCallStructHelper<R, P...>::Wrapper));
uintptr_t FunctionAddress = reinterpret_cast<uintptr_t>(Function);
@@ -182,8 +189,7 @@ protected:
template<typename R, typename... P>
void GenerateIndirectRuntimeCall(ARMEmitter::Register Reg) {
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(
&(vixl::aarch64::Simulator::RuntimeCallStructHelper<R, P...>::Wrapper));
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(&(vixl::aarch64::Simulator::RuntimeCallStructHelper<R, P...>::Wrapper));
hlt(vixl::aarch64::kIndirectRuntimeCallOpcode);
@@ -199,8 +205,8 @@ protected:
template<>
void GenerateIndirectRuntimeCall<float, __uint128_t>(ARMEmitter::Register Reg) {
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(
&(vixl::aarch64::Simulator::RuntimeCallStructHelper<float, __uint128_t>::Wrapper));
uintptr_t SimulatorWrapperAddress =
reinterpret_cast<uintptr_t>(&(vixl::aarch64::Simulator::RuntimeCallStructHelper<float, __uint128_t>::Wrapper));
hlt(vixl::aarch64::kIndirectRuntimeCallOpcode);
@@ -227,15 +233,17 @@ protected:
#ifdef VIXL_SIMULATOR
vixl::aarch64::Decoder SimDecoder;
vixl::aarch64::Simulator Simulator;
constexpr static size_t SimulatorStackSize = 8 * 1024 * 1024;
#endif
#ifdef VIXL_DISASSEMBLER
vixl::aarch64::Disassembler Disasm;
fextl::vector<char> DisasmBuffer;
constexpr static int DISASM_BUFFER_SIZE {256};
fextl::unique_ptr<vixl::aarch64::Disassembler> Disasm;
fextl::unique_ptr<vixl::aarch64::Decoder> DisasmDecoder;
FEX_CONFIG_OPT(Disassemble, DISASSEMBLE);
#endif
FEX_CONFIG_OPT(StaticRegisterAllocation, SRA);
};
}
} // namespace FEXCore::CPU
File diff suppressed because it is too large. Load diff
@@ -1,106 +0,0 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <cstddef>
#include <cstdint>
#include <cstring>
namespace FEXCore::ARMEmitter {
class Buffer {
public:
Buffer() {
SetBuffer(nullptr, 0);
}
Buffer(uint8_t* Base, uint64_t BaseSize) {
SetBuffer(Base, BaseSize);
}
void SetBuffer(uint8_t* Base, uint64_t BaseSize) {
BufferBase = Base;
CurrentOffset = BufferBase;
Size = BaseSize;
}
void dc8(uint8_t Data) {
decltype(Data) *Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void dc16(uint16_t Data) {
decltype(Data) *Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void dc32(uint32_t Data) {
decltype(Data) *Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void dc64(uint64_t Data) {
decltype(Data) *Memory = reinterpret_cast<decltype(Data)*>(CurrentOffset);
*Memory = Data;
CurrentOffset += sizeof(Data);
}
void EmitString(const char *String) {
const auto StringLength = strlen(String);
memcpy(CurrentOffset, String, StringLength);
CurrentOffset += StringLength;
}
void Align() {
// Align the buffer to instruction size
auto CurrentAlignment = reinterpret_cast<uint64_t>(CurrentOffset) & 0b11;
if (!CurrentAlignment) {
return;
}
CurrentOffset += 4 - CurrentAlignment;
}
template<typename T>
T GetCursorAddress() const {
return reinterpret_cast<T>(CurrentOffset);
}
static void ClearICache(void* Begin, std::size_t Length) {
__builtin___clear_cache(static_cast<char*>(Begin), static_cast<char*>(Begin) + Length);
}
size_t GetCursorOffset() const {
return static_cast<size_t>(CurrentOffset - BufferBase);
}
uint8_t *GetBufferBase() const {
return BufferBase;
}
void CursorIncrement(size_t Size) {
CurrentOffset += Size;
}
void SetCursorOffset(size_t Offset) {
CurrentOffset = BufferBase + Offset;
}
uint64_t GetBufferSize() const {
return Size;
}
template<typename T>
size_t GetCursorOffsetFromAddress(const T* Address) const {
return static_cast<size_t>(reinterpret_cast<const uint8_t*>(Address) - BufferBase);
}
protected:
void ResetBuffer() {
CurrentOffset = BufferBase;
}
uint8_t* BufferBase;
uint8_t* CurrentOffset;
uint64_t Size;
};
}
@@ -1,834 +0,0 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/ArchHelpers/CodeEmitter/Buffer.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Registers.h"
#include <FEXCore/Utils/BitUtils.h>
#include <FEXCore/Utils/CompilerDefs.h>
#include <FEXCore/Utils/EnumUtils.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/MathUtils.h>
#include <FEXCore/fextl/vector.h>
#include <aarch64/assembler-aarch64.h>
#include <array>
#include <cstdint>
#include <utility>
#include <type_traits>
/*
* Welcome to FEX-Emu's custom AArch64 emitter.
* This was written specifically to avoid the performance cost of the vixl emitter.
*
* There are some specific design constraints in this design to target a couple features:
* - High performance
* - Low CPU cache performance hit
* - Significantly reduced code footprint
* - Low number of branches
*
* These requirements are mostly achieved by removing a bunch of developer conveniences
* that vixl provides. The developer needs to take a lot of care to not shoot themselves in the foot.
*
* Misc design decisions:
* - Registers are encoded as basic uint32_t enums.
* - Converting between different registers is zero-cost.
* - Passing around as arguments are as cheap as registers
* - Contrast to vixl where every register requires living on the stack.
* - Registers can get encoded in to instructions with a simple `BFM` instruction.
*
* - Instructions are very simply emitted, allowing direct inlining most of the time.
* - These are simple enough that multiple back-to-back instructions get optimized to 128-bit load-store operations.
* - Contrast to vixl where pretty much no instruction emitter gets inlined.
*
* - Instruction emitters are /mostly/ unsized. Most instructions take a size argument first, which gets encoded
* directly in to the instruction.
* - Contrast to vixl where the register arguments are how the instructions determine operating size.
* - Size argument allows FEX to use `CSEL` to select a size at runtime, instead of branching.
* - Some instructions are explicitly sized based on register type. Read comments in the respective `inl` files to
* see why.
* Some scalar/vector operations are an example of this.
*
* - Almost zero helper functions.
* - Primary exception to this rule is load-store operations. These will use a helper to make
* it easier to select the correct load-store instruction. Mostly because these are a nightmare selecting
* the right instruction.
*/
namespace FEXCore::ARMEmitter {
/*
* This `Size` enum is used for most ALU operations.
* These follow the AArch64 encoding style in most cases.
*/
enum class Size : uint32_t {
i32Bit = 0,
i64Bit,
};
// This allows us to get the `Size` enum in bits.
[[nodiscard]]
constexpr size_t RegSizeInBits(Size size) {
return size_t{32} << FEXCore::ToUnderlying(size);
}
/* This `SubRegSize` enum is used for most ASIMD operations.
* These follow the AArch64 encoding style in most cases.
*/
enum class SubRegSize : uint32_t {
i8Bit = 0b00,
i16Bit = 0b01,
i32Bit = 0b10,
i64Bit = 0b11,
i128Bit = 0b100,
};
// This allows us to get the `SubRegSize` in bits.
[[nodiscard]]
constexpr size_t SubRegSizeInBits(SubRegSize size) {
return size_t{8} << FEXCore::ToUnderlying(size);
}
/* This `ScalarRegSize` enum is used for most scalar float
* operations.
*
* This is specifically duplicated from `SubRegSize` to have strongly
* typed functions.
*
* `ScalarRegSize` specifically doesn't have `i128Bit` because scalar operations
* can't operate at 128-bit.
*/
enum class ScalarRegSize : uint32_t {
i8Bit = 0b00,
i16Bit = 0b01,
i32Bit = 0b10,
i64Bit = 0b11,
};
// This allows us to get the `ScalarRegSize` in bits.
[[nodiscard]]
constexpr size_t ScalarRegSizeInBits(ScalarRegSize size) {
return size_t{8} << FEXCore::ToUnderlying(size);
}
/* This `VectorRegSizePair` union allows us to have an overlapping type
* to select a scalar operation or a vector depending on which operation
* we pass in.
* Useful in FEX's vector operations that behave as scalar or vector
* depending on various factors. But since the operation will have the sa,e
* element size, we want to choose the operation more easily
*/
union VectorRegSizePair {
ScalarRegSize Scalar;
SubRegSize Vector;
};
// This allows us to create a `VectorRegSizePair` union.
[[nodiscard]]
constexpr VectorRegSizePair ToVectorSizePair(SubRegSize size) {
return VectorRegSizePair {.Vector = size};
}
[[nodiscard]]
constexpr VectorRegSizePair ToVectorSizePair(ScalarRegSize size) {
return VectorRegSizePair {.Scalar = size};
}
// This `ShiftType` enum is used for ALU shift-register encoded instructions.
enum class ShiftType : uint32_t {
LSL = 0,
LSR,
ASR,
ROR,
};
// This `ExtendedType` enum is used for ALU extended-register encoded instructions.
enum class ExtendedType : uint32_t {
UXTB = 0b000,
UXTH = 0b001,
UXTW = 0b010,
UXTX = 0b011,
SXTB = 0b100,
SXTH = 0b101,
SXTW = 0b110,
SXTX = 0b111,
LSL_32 = UXTW,
LSL_64 = UXTX,
};
// This `Condition` enum is used for various conditional instructions.
enum class Condition : uint32_t {
// Meaning: Int - Float
CC_EQ = 0, // Equal - Equal
CC_NE, // Not Eq - Not Eq or unordered
CC_CS, // Carry set - Greater than, equal, or unordered
CC_CC, // Carry clear - Less than
CC_MI, // Minus/Negative - Less than
CC_PL, // Plus, positive or zero - GT, equal, or unordered
CC_VS, // Overflow - Unordered
CC_VC, // No Overflow - Ordered
CC_HI, // Unsigned higher - GT, or unordered
CC_LS, // Unsigned lower or same - LT or EQ
CC_GE, // Signed GT or EQ - GT or EQ
CC_LT, // Signed LT - LT or Unordered
CC_GT, // Signed GT - GT
CC_LE, // Signed LT or EQ - LT, EQ, or Unordered
CC_AL, // Always - Always
CC_NV, // Always - Always
// Aliases
CC_HS = CC_CS,
CC_LO = CC_CC,
};
/*
* This `StatusFlags` enum is used for conditional compare encoded instructions.
* These directly encode to the `nzcv` flags.
*/
enum class StatusFlags : uint32_t {
None = 0,
Flag_V = 0b0001,
Flag_C = 0b0010,
Flag_Z = 0b0100,
Flag_N = 0b1000,
Flag_NZCV = Flag_N | Flag_Z | Flag_C | Flag_V,
};
/*
* This `IndexType` enum is used for load-store instructions.
* Not all load-store instructions use this, so the user needs to be careful.
*/
enum class IndexType {
POST,
OFFSET,
PRE,
UNPRIVILEGED,
};
// Used with adr and scalar + vector load/store variants to denote
// a modifier operation.
enum class SVEModType : uint8_t {
MOD_UXTW,
MOD_SXTW,
MOD_LSL,
MOD_NONE,
};
/* This `SVEMemOperand` class is used for the helper SVE load-store instructions.
* Load-store instructions are quite expressive, so having a helper that handles these differences is worth it.
*/
class SVEMemOperand final {
public:
enum class Type {
ScalarPlusScalar,
ScalarPlusImm,
ScalarPlusVector,
VectorPlusImm,
};
SVEMemOperand(XRegister rn, XRegister rm = XReg::zr)
: rn {rn}
, MemType{Type::ScalarPlusScalar}
, MetaType {
.ScalarScalarType {
.rm = rm,
}
} {}
SVEMemOperand(XRegister rn, int32_t imm = 0)
: rn {rn}
, MemType{Type::ScalarPlusImm}
, MetaType {
.ScalarImmType {
.Imm = imm,
}
} {}
SVEMemOperand(XRegister rn, ZRegister zm, SVEModType mod = SVEModType::MOD_NONE, uint8_t scale = 0)
: rn{rn}
, MemType{Type::ScalarPlusVector}
, MetaType {
.ScalarVectorType {
.zm = zm,
.mod = mod,
.scale = scale,
}
} {}
SVEMemOperand(ZRegister zn, uint32_t imm)
: rn{Register{zn.Idx()}}
, MemType{Type::VectorPlusImm}
, MetaType {
.VectorImmType{
.Imm = imm,
}
} {}
[[nodiscard]] bool IsScalarPlusScalar() const {
return MemType == Type::ScalarPlusScalar;
}
[[nodiscard]] bool IsScalarPlusImm() const {
return MemType == Type::ScalarPlusImm;
}
[[nodiscard]] bool IsScalarPlusVector() const {
return MemType == Type::ScalarPlusVector;
}
[[nodiscard]] bool IsVectorPlusImm() const {
return MemType == Type::VectorPlusImm;
}
union Data {
struct {
Register rm;
} ScalarScalarType;
struct {
int32_t Imm;
} ScalarImmType;
struct {
ZRegister zm;
SVEModType mod;
uint8_t scale;
} ScalarVectorType;
struct {
// rn will be a ZRegister
uint32_t Imm;
} VectorImmType;
};
Register rn;
Type MemType;
Data MetaType;
};
/* This `ExtendedMemOperand` class is used for the helper load-store instructions.
* Load-store instructions are quite expressive, so having a helper that handles these differences is worth it.
*/
class ExtendedMemOperand final {
public:
ExtendedMemOperand(XRegister rn, XRegister rm = XReg::zr, ExtendedType Option = ExtendedType::LSL_64, uint32_t Shift = 0)
: rn {rn}
, MetaType {
.ExtendedType {
.Header = { .MemType = TYPE_EXTENDED },
.rm = rm,
.Option = Option,
.Shift = Shift,
}
} {}
ExtendedMemOperand(XRegister rn, IndexType Index = IndexType::OFFSET, int32_t Imm = 0)
: rn {rn}
, MetaType {
.ImmType {
.Header = { .MemType = TYPE_IMM },
.Index = Index,
.Imm = Imm,
}
} {}
Register rn;
enum Type {
TYPE_EXTENDED,
TYPE_IMM,
};
struct HeaderStruct {
Type MemType;
};
union {
HeaderStruct Header;
struct {
HeaderStruct Header;
Register rm;
ExtendedType Option;
uint32_t Shift;
} ExtendedType;
struct {
HeaderStruct Header;
IndexType Index;
int32_t Imm;
} ImmType;
} MetaType;
};
template<uint32_t op0, uint32_t op1, uint32_t CRn, uint32_t CRm, uint32_t op2>
constexpr uint32_t GenSystemReg() {
return op0 << 19 |
op1 << 16 |
CRn << 12 |
CRm << 8 |
op2 << 5;
};
// This `SystemRegister` enum is used for the mrs/msr instructions.
enum class SystemRegister : uint32_t {
CTR_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b001>(),
DCZID_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b111>(),
TPIDR_EL0 = GenSystemReg<0b11, 0b011, 0b1101, 0b0000, 0b010>(),
RNDR = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b000>(),
RNDRRS = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b001>(),
NZCV = GenSystemReg<0b11, 0b011, 0b0100, 0b0010, 0b000>(),
FPCR = GenSystemReg<0b11, 0b011, 0b0100, 0b0100, 0b000>(),
CNTFRQ_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b000>(),
CNTVCT_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b010>(),
};
template<uint32_t op1, uint32_t CRm, uint32_t op2>
constexpr uint32_t GenDCReg() {
return op1 << 16 |
CRm << 8 |
op2 << 5;
};
// This `DataCacheOperation` enum is used for the dc instruction.
enum class DataCacheOperation : uint32_t {
IVAC = GenDCReg<0b000, 0b0110, 0b001>(),
ISW = GenDCReg<0b000, 0b0110, 0b010>(),
CSW = GenDCReg<0b000, 0b1010, 0b010>(),
CISW = GenDCReg<0b000, 0b1110, 0b010>(),
ZVA = GenDCReg<0b011, 0b0100, 0b001>(),
CVAC = GenDCReg<0b011, 0b1010, 0b001>(),
CVAU = GenDCReg<0b011, 0b1011, 0b001>(),
CIVAC = GenDCReg<0b011, 0b1110, 0b001>(),
// MTE2
IGVAC = GenDCReg<0b000, 0b0110, 0b011>(),
IGSW = GenDCReg<0b000, 0b0110, 0b100>(),
IGDVAC = GenDCReg<0b000, 0b0110, 0b101>(),
IGDSW = GenDCReg<0b000, 0b0110, 0b110>(),
CGSW = GenDCReg<0b000, 0b1010, 0b100>(),
CGDSW = GenDCReg<0b000, 0b1010, 0b110>(),
CIGSW = GenDCReg<0b000, 0b1110, 0b100>(),
CIGDSW = GenDCReg<0b000, 0b1110, 0b110>(),
// MTE
GVA = GenDCReg<0b011, 0b0100, 0b011>(),
GZVA = GenDCReg<0b011, 0b0100, 0b100>(),
CGVAC = GenDCReg<0b011, 0b1010, 0b011>(),
CGDVAC = GenDCReg<0b011, 0b1010, 0b101>(),
CGVAP = GenDCReg<0b011, 0b1100, 0b011>(),
CGDVAP = GenDCReg<0b011, 0b1100, 0b101>(),
CGVADP = GenDCReg<0b011, 0b1101, 0b011>(),
CGDVADP = GenDCReg<0b011, 0b1101, 0b101>(),
CIGVAC = GenDCReg<0b011, 0b1110, 0b011>(),
CIGDVAC = GenDCReg<0b011, 0b1110, 0b101>(),
// DPB
CVAP = GenDCReg<0b011, 0b1100, 0b001>(),
// DPB2
CVADP = GenDCReg<0b011, 0b1101, 0b001>(),
};
template<uint32_t CRm, uint32_t op2>
constexpr uint32_t GenHintBarrierReg() {
return CRm << 8 |
op2 << 5;
}
// This `HintRegister` enum is used for the hint instruction.
enum class HintRegister : uint32_t {
NOP = GenHintBarrierReg<0b0000, 0b000>(),
YIELD = GenHintBarrierReg<0b0000, 0b001>(),
WFE = GenHintBarrierReg<0b0000, 0b010>(),
WFI = GenHintBarrierReg<0b0000, 0b011>(),
SEV = GenHintBarrierReg<0b0000, 0b100>(),
SEVL = GenHintBarrierReg<0b0000, 0b101>(),
DGH = GenHintBarrierReg<0b0000, 0b110>(),
CSDB = GenHintBarrierReg<0b0010, 0b100>(),
};
// This `BarrierRegister` enum is used for the various barrier instructions.
enum class BarrierRegister : uint32_t {
CLREX = GenHintBarrierReg<0b0000, 0b010>(),
TCOMMIT = GenHintBarrierReg<0b0000, 0b011>(),
DSB = GenHintBarrierReg<0b0000, 0b100>(),
DMB = GenHintBarrierReg<0b0000, 0b101>(),
ISB = GenHintBarrierReg<0b0000, 0b110>(),
SB = GenHintBarrierReg<0b0000, 0b111>(),
};
// This `BarrierScope` enum is used for the dsb/dmb instructions.
enum class BarrierScope : uint32_t {
// Outer shareable
OSHLD = 0b0001,
OSHST = 0b0010,
OSH = 0b0011,
// Non shareable
NSHLD = 0b0101,
NSHST = 0b0110,
NSH = 0b0111,
// Inner shareable
ISHLD = 0b1001,
ISHST = 0b1010,
ISH = 0b1011,
// Full System visibility
LD = 0b1101,
ST = 0b1110,
SY = 0b1111,
};
// This `Prefetch` enum is used for prefetch instructions.
enum class Prefetch : uint32_t {
// Prefetch for load
PLDL1KEEP = 0b00000,
PLDL1STRM = 0b00001,
PLDL2KEEP = 0b00010,
PLDL2STRM = 0b00011,
PLDL3KEEP = 0b00100,
PLDL3STRM = 0b00101,
// Preload instructions
PLIL1KEEP = 0b01000,
PLIL1STRM = 0b01001,
PLIL2KEEP = 0b01010,
PLIL2STRM = 0b01011,
PLIL3KEEP = 0b01100,
PLIL3STRM = 0b01101,
// Preload for store
PSTL1KEEP = 0b10000,
PSTL1STRM = 0b10001,
PSTL2KEEP = 0b10010,
PSTL2STRM = 0b10011,
PSTL3KEEP = 0b10100,
PSTL3STRM = 0b10101,
};
// This `PredicatePattern` enun is used for some SVE instructions.
enum class PredicatePattern : uint32_t {
SVE_POW2 = 0b00000,
SVE_VL1 = 0b00001,
SVE_VL2 = 0b00010,
SVE_VL3 = 0b00011,
SVE_VL4 = 0b00100,
SVE_VL5 = 0b00101,
SVE_VL6 = 0b00110,
SVE_VL7 = 0b00111,
SVE_VL8 = 0b01000,
SVE_VL16 = 0b01001,
SVE_VL32 = 0b01010,
SVE_VL64 = 0b01011,
SVE_VL128 = 0b01100,
SVE_VL256 = 0b01101,
SVE_MUL4 = 0b11101,
SVE_MUL3 = 0b11110,
SVE_ALL = 0b11111,
};
// Used with SVE FP immediate arithmetic instructions
enum class SVEFAddSubImm : uint32_t {
_0_5,
_1_0,
};
enum class SVEFMulImm : uint32_t {
_0_5,
_2_0,
};
enum class SVEFMaxMinImm : uint32_t {
_0_0,
_1_0,
};
/* This `BackwardLabel` struct used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is logically `below` an instruction that uses it.
* Which means that a branch would jump backwards.
*/
struct BackwardLabel {
uint8_t *Location{};
};
/* This `ForwardLabel` struct used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is logically `above` an instruction that uses it.
* Which means that a branch would jump forwards.
*
* This can be bound to multiple instructions, so it needs a vector for each bind instruction type.
*/
struct ForwardLabel {
struct Instructions {
enum class InstType {
ADR,
ADRP,
B,
BC,
TEST_BRANCH,
RELATIVE_LOAD,
LONG_ADDRESS_GEN,
};
uint8_t *Location{};
InstType Type;
};
fextl::vector<Instructions> Insts{};
};
/* This `BiDirectionalLabel` struct used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is in either direction of an instruction that uses it.
* Which means a branch could jump backwards or forwards depending on situation.
*/
struct BiDirectionalLabel {
BackwardLabel Backward;
ForwardLabel Forward;
};
// Some FCMA ASIMD instructions support a rotation argument.
enum class Rotation : uint32_t {
ROTATE_0 = 0b00,
ROTATE_90 = 0b01,
ROTATE_180 = 0b10,
ROTATE_270 = 0b11,
};
// Concept for contraining some instructions to accept only an XRegister or WRegister.
// Particularly for operations that differ encodings depending on which one is used.
template <typename T>
concept IsXOrWRegister = std::is_same_v<T, XRegister> || std::is_same_v<T, WRegister>;
// Whether or not a given set of vector registers are sequential
// in increasing order as far as the register file is concerned (modulo its size)
//
// For example, a set of registers like:
//
// v1, v2, v3 and
// v31, v0, v1
//
// would both be considered sequential sequences, and some instructions in particular
// limit register lists to these kind of sequences.
//
template <typename T, typename... Args>
constexpr bool AreVectorsSequential(T first, const Args&... args) {
// Ensure we always have a pair of registers to compare against.
static_assert(sizeof...(args) >= 1, "Number of arguments must be greater than 1");
const auto fn = [](auto& lhs, const auto& rhs) {
const auto result = ((lhs.Idx() + 1) % 32) == rhs.Idx();
lhs = rhs;
return result;
};
return (fn(first, args) && ...);
}
// This is an emitter that is designed around the smallest code bloat as possible.
// Eschewing most developer convenience in order to keep code as small as possible.
// Choices:
// - Size of ops passed as an argument rather than template to let the compiler use csel instead of branching.
// - Registers are unsized so they can be passed in a GPR and not need conversion operations
class Emitter : public FEXCore::ARMEmitter::Buffer {
public:
Emitter() = default;
Emitter(uint8_t* Base, uint64_t BaseSize)
: Buffer (Base, BaseSize) {
}
// Bind a backward label to an address.
// Address that is bound is the current emitter location.
void Bind(BackwardLabel *Label) {
LOGMAN_THROW_AA_FMT(Label->Location == nullptr, "Trying to bind a label twice");
Label->Location = GetCursorAddress<uint8_t*>();
}
// Bind a forward label to a location.
// This walks all the instructions in the label's vector.
// Then backpatching all instructions that have used the label.
template<bool WarnAboutEmpty = false>
void Bind(ForwardLabel *Label) {
if constexpr (WarnAboutEmpty) {
LOGMAN_THROW_A_FMT(Label->Insts.empty() == false, "Binding forward label that didn't have any instructions using it");
}
uint8_t *CurrentAddress = GetCursorAddress<uint8_t*>();
for (const auto &Inst : Label->Insts) {
// Patch up the instructions
switch (Inst.Type) {
case ForwardLabel::Instructions::InstType::ADR: {
uint32_t *Instruction = reinterpret_cast<uint32_t*>(Inst.Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(IsADRRange(Imm), "Unscaled offset too large");
uint32_t InstMask = 0b11 << 29 | 0b1111'1111'1111'1111'111 << 5;
uint32_t Offset = static_cast<uint32_t>(Imm) & 0x3F'FFFF;
uint32_t Inst = *Instruction & ~InstMask;
Inst |= (Offset & 0b11) << 29;
Inst |= (Offset >> 2) << 5;
*Instruction = Inst;
break;
}
case ForwardLabel::Instructions::InstType::ADRP: {
uint32_t *Instruction = reinterpret_cast<uint32_t*>(Inst.Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(IsADRPRange(Imm) && IsADRPAligned(Imm), "Unscaled offset too large");
Imm >>= 12;
uint32_t InstMask = 0b11 << 29 | 0b1111'1111'1111'1111'111 << 5;
uint32_t Offset = static_cast<uint32_t>(Imm) & 0x3F'FFFF;
uint32_t Inst = *Instruction & ~InstMask;
Inst |= (Offset & 0b11) << 29;
Inst |= (Offset >> 2) << 5;
*Instruction = Inst;
break;
}
case ForwardLabel::Instructions::InstType::B: {
uint32_t *Instruction = reinterpret_cast<uint32_t*>(Inst.Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -134217728 && Imm <= 134217724 && ((Imm & 0b11) == 0), "Unscaled offset too large");
Imm >>= 2;
uint32_t InstMask = 0x3FF'FFFF;
uint32_t Offset = static_cast<uint32_t>(Imm) & InstMask;
uint32_t Inst = *Instruction & ~InstMask;
Inst |= Offset;
*Instruction = Inst;
break;
}
case ForwardLabel::Instructions::InstType::TEST_BRANCH: {
uint32_t *Instruction = reinterpret_cast<uint32_t*>(Inst.Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
Imm >>= 2;
uint32_t InstMask = 0x3FFF;
uint32_t Offset = static_cast<uint32_t>(Imm) & InstMask;
uint32_t Inst = *Instruction & ~(InstMask << 5);
Inst |= Offset << 5;
*Instruction = Inst;
break;
}
case ForwardLabel::Instructions::InstType::BC:
case ForwardLabel::Instructions::InstType::RELATIVE_LOAD: {
uint32_t *Instruction = reinterpret_cast<uint32_t*>(Inst.Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
Imm >>= 2;
uint32_t InstMask = 0x7'FFFF;
uint32_t Offset = static_cast<uint32_t>(Imm) & InstMask;
uint32_t Inst = *Instruction & ~(InstMask << 5);
Inst |= Offset << 5;
*Instruction = Inst;
break;
}
case ForwardLabel::Instructions::InstType::LONG_ADDRESS_GEN: {
uint32_t *Instructions = reinterpret_cast<uint32_t*>(Inst.Location);
int64_t ImmInstOne = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(&Instructions[0]);
int64_t ImmInstTwo = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(&Instructions[1]);
auto OriginalOffset = GetCursorOffset();
auto InstOffset = GetCursorOffsetFromAddress(Instructions);
SetCursorOffset(InstOffset);
// We encoded the destination register in to the first instruction space.
// Read it back.
ARMEmitter::Register DestReg(Instructions[0]);
if (IsADRRange(ImmInstTwo)) {
// If within ADR range from the second instruction, then we can emit NOP+ADR
nop();
adr(DestReg, static_cast<uint32_t>(ImmInstTwo) & 0x7FFF);
}
else if (IsADRPRange(ImmInstOne)) {
// If within ADRP range from the first instruction, then we are /definitely/ in range for the second instruction.
// First check if we are in non-offset range for second instruction.
if (IsADRPAligned(reinterpret_cast<uint64_t>(CurrentAddress))) {
// We can emit nop + adrp
nop();
adrp(DestReg, static_cast<uint32_t>(ImmInstTwo >> 12) & 0x7FFF);
}
else {
// Not aligned, need adrp + add
adrp(DestReg, static_cast<uint32_t>(ImmInstOne >> 12) & 0x7FFF);
add(ARMEmitter::Size::i64Bit, DestReg, DestReg, ImmInstOne & 0xFFF);
}
}
else {
LOGMAN_MSG_A_FMT("Unscaled offset is too large");
FEX_UNREACHABLE;
}
SetCursorOffset(OriginalOffset);
break;
}
default: LOGMAN_MSG_A_FMT("Unexpected inst type in label fixup");
}
}
}
// Bind a bidirectional location to a location.
// Binds both forwards and backwards depending on how the label was used.
void Bind(BiDirectionalLabel *Label) {
if (!Label->Backward.Location) {
Bind(&Label->Backward);
}
Bind<false>(&Label->Forward);
}
public:
// TODO: Implement SME when it matters.
#include "Interface/Core/ArchHelpers/CodeEmitter/ALUOps.inl"
#include "Interface/Core/ArchHelpers/CodeEmitter/BranchOps.inl"
#include "Interface/Core/ArchHelpers/CodeEmitter/LoadstoreOps.inl"
#include "Interface/Core/ArchHelpers/CodeEmitter/SystemOps.inl"
#include "Interface/Core/ArchHelpers/CodeEmitter/ScalarOps.inl"
#include "Interface/Core/ArchHelpers/CodeEmitter/ASIMDOps.inl"
#include "Interface/Core/ArchHelpers/CodeEmitter/SVEOps.inl"
private:
template<typename T>
uint32_t Encode_ra(T Reg) const {
return Reg.Idx() << 10;
}
uint32_t Encode_ra(uint32_t Reg) const {
return Reg << 10;
}
template<typename T>
uint32_t Encode_rt2(T Reg) const {
return Reg.Idx() << 10;
}
template<>
uint32_t Encode_rt2(uint32_t Reg) const {
return Reg << 10;
}
template<typename T>
uint32_t Encode_rm(T Reg) const {
return Reg.Idx() << 16;
}
uint32_t Encode_rm(uint32_t Reg) const {
return Reg << 16;
}
template<typename T>
uint32_t Encode_rs(T Reg) const {
return Reg.Idx() << 16;
}
uint32_t Encode_rs(uint32_t Reg) const {
return Reg << 16;
}
template<typename T>
uint32_t Encode_rn(T Reg) const {
return Reg.Idx() << 5;
}
uint32_t Encode_rn(uint32_t Reg) const {
return Reg << 5;
}
template<typename T>
uint32_t Encode_rd(T Reg) const {
return Reg.Idx();
}
uint32_t Encode_rd(uint32_t Reg) const {
return Reg;
}
template<typename T>
uint32_t Encode_rt(T Reg) const {
return Reg.Idx();
}
template<>
uint32_t Encode_rt(Prefetch Reg) const {
return FEXCore::ToUnderlying(Reg);
}
uint32_t Encode_rt(uint32_t Reg) const {
return Reg;
}
template<typename T>
uint32_t Encode_pd(T Reg) const {
return FEXCore::ToUnderlying(Reg);
}
};
}
File diff suppressed because it is too large. Load diff
@@ -6,48 +6,46 @@
#include <utility>
namespace FEXCore {
void BlockSamplingData::DumpBlockData() {
std::fstream Output;
Output.open("output.csv", std::fstream::out | std::fstream::binary);
void BlockSamplingData::DumpBlockData() {
std::fstream Output;
Output.open("output.csv", std::fstream::out | std::fstream::binary);
if (!Output.is_open())
return;
Output << "Entry, Min, Max, Total, Calls, Average" << std::endl;
for (auto it : SamplingMap) {
if (!it.second->TotalCalls)
continue;
Output << "0x" << std::hex << it.first
<< ", " << std::dec << it.second->Min
<< ", " << std::dec << it.second->Max
<< ", " << std::dec << it.second->TotalTime
<< ", " << std::dec << it.second->TotalCalls
<< ", " << std::dec << ((double)it.second->TotalTime / (double)it.second->TotalCalls)
<< std::endl;
}
Output.close();
LogMan::Msg::DFmt("Dumped {} blocks of sampling data", SamplingMap.size());
if (!Output.is_open()) {
return;
}
BlockSamplingData::BlockData *BlockSamplingData::GetBlockData(uint64_t RIP) {
auto it = SamplingMap.find(RIP);
if (it != SamplingMap.end()) {
return it->second;
}
BlockData *NewData = new BlockData{};
memset(NewData, 0, sizeof(BlockData));
NewData->Min = ~0ULL;
SamplingMap[RIP] = NewData;
return NewData;
}
Output << "Entry, Min, Max, Total, Calls, Average" << std::endl;
BlockSamplingData::~BlockSamplingData() {
DumpBlockData();
for (auto it : SamplingMap) {
delete it.second;
for (auto it : SamplingMap) {
if (!it.second->TotalCalls) {
continue;
}
SamplingMap.clear();
Output << "0x" << std::hex << it.first << ", " << std::dec << it.second->Min << ", " << std::dec << it.second->Max << ", " << std::dec
<< it.second->TotalTime << ", " << std::dec << it.second->TotalCalls << ", " << std::dec
<< ((double)it.second->TotalTime / (double)it.second->TotalCalls) << std::endl;
}
Output.close();
LogMan::Msg::DFmt("Dumped {} blocks of sampling data", SamplingMap.size());
}
BlockSamplingData::BlockData* BlockSamplingData::GetBlockData(uint64_t RIP) {
auto it = SamplingMap.find(RIP);
if (it != SamplingMap.end()) {
return it->second;
}
BlockData* NewData = new BlockData {};
memset(NewData, 0, sizeof(BlockData));
NewData->Min = ~0ULL;
SamplingMap[RIP] = NewData;
return NewData;
}
BlockSamplingData::~BlockSamplingData() {
DumpBlockData();
for (auto it : SamplingMap) {
delete it.second;
}
SamplingMap.clear();
}
} // namespace FEXCore
@@ -14,7 +14,7 @@ public:
uint64_t TotalCalls;
};
BlockData *GetBlockData(uint64_t RIP);
BlockData* GetBlockData(uint64_t RIP);
~BlockSamplingData();
void DumpBlockData();
@@ -22,4 +22,4 @@ public:
private:
std::unordered_map<uint64_t, BlockData*> SamplingMap;
};
}
} // namespace FEXCore
+360 -250
View File
@@ -2,282 +2,392 @@
#include "FEXCore/IR/IR.h"
#include "FEXCore/Utils/AllocatorHooks.h"
#include "Interface/Context/Context.h"
#include "Interface/Core/CPUBackend.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include <FEXCore/Core/CPUBackend.h>
#ifndef _WIN32
#include <sys/prctl.h>
#endif
namespace FEXCore {
namespace CPU {
constexpr static uint64_t NamedVectorConstants[FEXCore::IR::NamedVectorConstant::NAMED_VECTOR_CONST_POOL_MAX][2] = {
{0x0003'0002'0001'0000ULL, 0x0007'0006'0005'0004ULL}, // NAMED_VECTOR_INCREMENTAL_U16_INDEX
{0x000B'000A'0009'0008ULL, 0x000F'000E'000D'000CULL}, // NAMED_VECTOR_INCREMENTAL_U16_INDEX_UPPER
{0x0000'0000'8000'0000ULL, 0x0000'0000'8000'0000ULL}, // NAMED_VECTOR_PADDSUBPS_INVERT
{0x0000'0000'8000'0000ULL, 0x0000'0000'8000'0000ULL}, // NAMED_VECTOR_PADDSUBPS_INVERT_UPPER
{0x8000'0000'0000'0000ULL, 0x0000'0000'0000'0000ULL}, // NAMED_VECTOR_PADDSUBPD_INVERT
{0x8000'0000'0000'0000ULL, 0x0000'0000'0000'0000ULL}, // NAMED_VECTOR_PADDSUBPD_INVERT_UPPER
{0x0000'0001'0000'0000ULL, 0x0000'0003'0000'0002ULL}, // NAMED_VECTOR_MOVMSKPS_SHIFT
{0x040B'0E01'0B0E'0104ULL, 0x0C03'0609'0306'090CULL}, // NAMED_VECTOR_AESKEYGENASSIST_SWIZZLE
};
constexpr static auto PSHUFLW_LUT {
[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// Expectation for this LUT is to simulate PSHUFLW with ARM's TBL (single register) instruction
// PSHUFLW behaviour:
// 16-bit words in [63:48], [47:32], [31:16], [15:0] are selected using the 8-bit Index.
// For 128-bit PSHUFLW, bits [127:64] are identity copied.
constexpr uint64_t IdentityCopyUpper = 0x0f'0e'0d'0c'0b'0a'09'08;
std::array<LUTType, 256> TotalLUT{};
uint64_t WordSelection[4] = {
0x01'00,
0x03'02,
0x05'04,
0x07'06,
};
for (size_t i = 0; i < 256; ++i) {
auto &LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
LUT.Val[0] =
(WordSelection[Word0] << 0) |
(WordSelection[Word1] << 16) |
(WordSelection[Word2] << 32) |
(WordSelection[Word3] << 48);
LUT.Val[1] = IdentityCopyUpper;
}
return TotalLUT;
}()
};
constexpr static auto PSHUFHW_LUT {
[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// Expectation for this LUT is to simulate PSHUFHW with ARM's TBL (single register) instruction
// PSHUFHW behaviour:
// 16-bit words in [127:112], [111:96], [95:80], [79:64] are selected using the 8-bit Index.
// Incoming words come from bits [127:64] of the source.
// Bits [63:0] are identity copied.
constexpr uint64_t IdentityCopyLower = 0x07'06'05'04'03'02'01'00;
std::array<LUTType, 256> TotalLUT{};
uint64_t WordSelection[4] = {
0x09'08,
0x0b'0a,
0x0d'0c,
0x0f'0e,
};
for (size_t i = 0; i < 256; ++i) {
auto &LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
LUT.Val[0] = IdentityCopyLower;
LUT.Val[1] =
(WordSelection[Word0] << 0) |
(WordSelection[Word1] << 16) |
(WordSelection[Word2] << 32) |
(WordSelection[Word3] << 48);
}
return TotalLUT;
}()
};
constexpr static auto PSHUFD_LUT {
[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// Expectation for this LUT is to simulate PSHUFD with ARM's TBL (single register) instruction
// PSHUFD behaviour:
// 32-bit words in [127:96], [95:64], [63:32], [31:0] are selected using the 8-bit Index.
std::array<LUTType, 256> TotalLUT{};
uint64_t WordSelection[4] = {
0x03'02'01'00,
0x07'06'05'04,
0x0b'0a'09'08,
0x0f'0e'0d'0c,
};
for (size_t i = 0; i < 256; ++i) {
auto &LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
LUT.Val[0] =
(WordSelection[Word0] << 0) |
(WordSelection[Word1] << 32);
LUT.Val[1] =
(WordSelection[Word2] << 0) |
(WordSelection[Word3] << 32);
}
return TotalLUT;
}()
};
constexpr static auto SHUFPS_LUT {
[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// 32-bit words in [127:96], [95:64], [63:32], [31:0] are selected using the 8-bit Index.
// Expectation for this LUT is to simulate SHUFPS with ARM's TBL (two register) instruction.
// SHUFPS behaviour:
// Two 32-bits words from each source are selected from each source in the lower and upper halves of the 128-bit destination.
// Dest[31:0] = Src1[<Word0>]
// Dest[63:32] = Src1[<Word1>]
// Dest[95:64] = Src2[<Word2>]
// Dest[127:96] = Src2[<Word3>]
std::array<LUTType, 256> TotalLUT{};
const uint64_t WordSelectionSrc1[4] = {
0x03'02'01'00,
0x07'06'05'04,
0x0b'0a'09'08,
0x0f'0e'0d'0c,
constexpr static uint64_t NamedVectorConstants[FEXCore::IR::NamedVectorConstant::NAMED_VECTOR_CONST_POOL_MAX][2] = {
{0x0003'0002'0001'0000ULL, 0x0007'0006'0005'0004ULL}, // NAMED_VECTOR_INCREMENTAL_U16_INDEX
{0x000B'000A'0009'0008ULL, 0x000F'000E'000D'000CULL}, // NAMED_VECTOR_INCREMENTAL_U16_INDEX_UPPER
{0x0000'0000'8000'0000ULL, 0x0000'0000'8000'0000ULL}, // NAMED_VECTOR_PADDSUBPS_INVERT
{0x0000'0000'8000'0000ULL, 0x0000'0000'8000'0000ULL}, // NAMED_VECTOR_PADDSUBPS_INVERT_UPPER
{0x8000'0000'0000'0000ULL, 0x0000'0000'0000'0000ULL}, // NAMED_VECTOR_PADDSUBPD_INVERT
{0x8000'0000'0000'0000ULL, 0x0000'0000'0000'0000ULL}, // NAMED_VECTOR_PADDSUBPD_INVERT_UPPER
{0x8000'0000'0000'0000ULL, 0x8000'0000'0000'0000ULL}, // NAMED_VECTOR_PSUBADDPS_INVERT
{0x8000'0000'0000'0000ULL, 0x8000'0000'0000'0000ULL}, // NAMED_VECTOR_PSUBADDPS_INVERT_UPPER
{0x0000'0000'0000'0000ULL, 0x8000'0000'0000'0000ULL}, // NAMED_VECTOR_PSUBADDPD_INVERT
{0x0000'0000'0000'0000ULL, 0x8000'0000'0000'0000ULL}, // NAMED_VECTOR_PSUBADDPD_INVERT_UPPER
{0x0000'0001'0000'0000ULL, 0x0000'0003'0000'0002ULL}, // NAMED_VECTOR_MOVMSKPS_SHIFT
{0x040B'0E01'0B0E'0104ULL, 0x0C03'0609'0306'090CULL}, // NAMED_VECTOR_AESKEYGENASSIST_SWIZZLE
{0x0706'0504'FFFF'FFFFULL, 0xFFFF'FFFF'0B0A'0908ULL}, // NAMED_VECTOR_BLENDPS_0110B
{0x0706'0504'0302'0100ULL, 0xFFFF'FFFF'0B0A'0908ULL}, // NAMED_VECTOR_BLENDPS_0111B
{0xFFFF'FFFF'0302'0100ULL, 0x0F0E'0D0C'FFFF'FFFFULL}, // NAMED_VECTOR_BLENDPS_1001B
{0x0706'0504'0302'0100ULL, 0x0F0E'0D0C'FFFF'FFFFULL}, // NAMED_VECTOR_BLENDPS_1011B
{0xFFFF'FFFF'0302'0100ULL, 0x0F0E'0D0C'0B0A'0908ULL}, // NAMED_VECTOR_BLENDPS_1101B
{0x0706'0504'FFFF'FFFFULL, 0x0F0E'0D0C'0B0A'0908ULL}, // NAMED_VECTOR_BLENDPS_1110B
{0x8040'2010'0804'0201ULL, 0x8040'2010'0804'0201ULL}, // NAMED_VECTOR_MOVMASKB
{0x8040'2010'0804'0201ULL, 0x8040'2010'0804'0201ULL}, // NAMED_VECTOR_MOVMASKB_UPPER
{0x8000'0000'0000'0000ULL, 0x0000'0000'0000'3FFFULL}, // NAMED_VECTOR_X87_ONE
{0xD49A'784B'CD1B'8AFEULL, 0x0000'0000'0000'4000ULL}, // NAMED_VECTOR_X87_LOG2_10
{0xB8AA'3B29'5C17'F0BCULL, 0x0000'0000'0000'3FFFULL}, // NAMED_VECTOR_X87_LOG2_E
{0xC90F'DAA2'2168'C235ULL, 0x0000'0000'0000'4000ULL}, // NAMED_VECTOR_X87_PI
{0x9A20'9A84'FBCF'F799ULL, 0x0000'0000'0000'3FFDULL}, // NAMED_VECTOR_X87_LOG10_2
{0xB172'17F7'D1CF'79ACULL, 0x0000'0000'0000'3FFEULL}, // NAMED_VECTOR_X87_LOG_2
};
// Src2 needs to offset each byte index by 16-bytes to pull from the second source.
const uint64_t WordSelectionSrc2[4] = {
0x03'02'01'00 + (0x10101010),
0x07'06'05'04 + (0x10101010),
0x0b'0a'09'08 + (0x10101010),
0x0f'0e'0d'0c + (0x10101010),
};
constexpr static auto PSHUFLW_LUT {[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// Expectation for this LUT is to simulate PSHUFLW with ARM's TBL (single register) instruction
// PSHUFLW behaviour:
// 16-bit words in [63:48], [47:32], [31:16], [15:0] are selected using the 8-bit Index.
// For 128-bit PSHUFLW, bits [127:64] are identity copied.
constexpr uint64_t IdentityCopyUpper = 0x0f'0e'0d'0c'0b'0a'09'08;
std::array<LUTType, 256> TotalLUT {};
uint64_t WordSelection[4] = {
0x01'00,
0x03'02,
0x05'04,
0x07'06,
};
for (size_t i = 0; i < 256; ++i) {
auto& LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
for (size_t i = 0; i < 256; ++i) {
auto &LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
LUT.Val[0] = (WordSelection[Word0] << 0) | (WordSelection[Word1] << 16) | (WordSelection[Word2] << 32) | (WordSelection[Word3] << 48);
LUT.Val[0] =
(WordSelectionSrc1[Word0] << 0) |
(WordSelectionSrc1[Word1] << 32);
LUT.Val[1] = IdentityCopyUpper;
}
return TotalLUT;
}()};
LUT.Val[1] =
(WordSelectionSrc2[Word2] << 0) |
(WordSelectionSrc2[Word3] << 32);
}
return TotalLUT;
}()
};
constexpr static auto PSHUFHW_LUT {[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// Expectation for this LUT is to simulate PSHUFHW with ARM's TBL (single register) instruction
// PSHUFHW behaviour:
// 16-bit words in [127:112], [111:96], [95:80], [79:64] are selected using the 8-bit Index.
// Incoming words come from bits [127:64] of the source.
// Bits [63:0] are identity copied.
constexpr uint64_t IdentityCopyLower = 0x07'06'05'04'03'02'01'00;
std::array<LUTType, 256> TotalLUT {};
uint64_t WordSelection[4] = {
0x09'08,
0x0b'0a,
0x0d'0c,
0x0f'0e,
};
for (size_t i = 0; i < 256; ++i) {
auto& LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
CPUBackend::CPUBackend(FEXCore::Core::InternalThreadState *ThreadState, size_t InitialCodeSize, size_t MaxCodeSize)
: ThreadState(ThreadState), InitialCodeSize(InitialCodeSize), MaxCodeSize(MaxCodeSize) {
LUT.Val[0] = IdentityCopyLower;
auto &Common = ThreadState->CurrentFrame->Pointers.Common;
LUT.Val[1] = (WordSelection[Word0] << 0) | (WordSelection[Word1] << 16) | (WordSelection[Word2] << 32) | (WordSelection[Word3] << 48);
}
return TotalLUT;
}()};
// Initialize named vector constants.
for (size_t i = 0; i < FEXCore::IR::NamedVectorConstant::NAMED_VECTOR_CONST_POOL_MAX; ++i) {
Common.NamedVectorConstantPointers[i] = reinterpret_cast<uint64_t>(NamedVectorConstants[i]);
}
constexpr static auto PSHUFD_LUT {[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// Expectation for this LUT is to simulate PSHUFD with ARM's TBL (single register) instruction
// PSHUFD behaviour:
// 32-bit words in [127:96], [95:64], [63:32], [31:0] are selected using the 8-bit Index.
std::array<LUTType, 256> TotalLUT {};
uint64_t WordSelection[4] = {
0x03'02'01'00,
0x07'06'05'04,
0x0b'0a'09'08,
0x0f'0e'0d'0c,
};
for (size_t i = 0; i < 256; ++i) {
auto& LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
// Copy named vector constants.
memcpy(Common.NamedVectorConstants, NamedVectorConstants, sizeof(NamedVectorConstants));
LUT.Val[0] = (WordSelection[Word0] << 0) | (WordSelection[Word1] << 32);
// Initialize Indexed named vector constants.
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_PSHUFLW] = reinterpret_cast<uint64_t>(PSHUFLW_LUT.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_PSHUFHW] = reinterpret_cast<uint64_t>(PSHUFHW_LUT.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_PSHUFD] = reinterpret_cast<uint64_t>(PSHUFD_LUT.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_SHUFPS] = reinterpret_cast<uint64_t>(SHUFPS_LUT.data());
LUT.Val[1] = (WordSelection[Word2] << 0) | (WordSelection[Word3] << 32);
}
return TotalLUT;
}()};
constexpr static auto SHUFPS_LUT {[]() consteval {
struct LUTType {
uint64_t Val[2];
};
// 32-bit words in [127:96], [95:64], [63:32], [31:0] are selected using the 8-bit Index.
// Expectation for this LUT is to simulate SHUFPS with ARM's TBL (two register) instruction.
// SHUFPS behaviour:
// Two 32-bits words from each source are selected from each source in the lower and upper halves of the 128-bit destination.
// Dest[31:0] = Src1[<Word0>]
// Dest[63:32] = Src1[<Word1>]
// Dest[95:64] = Src2[<Word2>]
// Dest[127:96] = Src2[<Word3>]
std::array<LUTType, 256> TotalLUT {};
const uint64_t WordSelectionSrc1[4] = {
0x03'02'01'00,
0x07'06'05'04,
0x0b'0a'09'08,
0x0f'0e'0d'0c,
};
// Src2 needs to offset each byte index by 16-bytes to pull from the second source.
const uint64_t WordSelectionSrc2[4] = {
0x03'02'01'00 + (0x10101010),
0x07'06'05'04 + (0x10101010),
0x0b'0a'09'08 + (0x10101010),
0x0f'0e'0d'0c + (0x10101010),
};
for (size_t i = 0; i < 256; ++i) {
auto& LUT = TotalLUT[i];
const auto Word0 = (i >> 0) & 0b11;
const auto Word1 = (i >> 2) & 0b11;
const auto Word2 = (i >> 4) & 0b11;
const auto Word3 = (i >> 6) & 0b11;
LUT.Val[0] = (WordSelectionSrc1[Word0] << 0) | (WordSelectionSrc1[Word1] << 32);
LUT.Val[1] = (WordSelectionSrc2[Word2] << 0) | (WordSelectionSrc2[Word3] << 32);
}
return TotalLUT;
}()};
constexpr static auto DPPS_MASK {[]() consteval {
struct LUTType {
uint32_t Val[4];
};
std::array<LUTType, 16> TotalLUT {};
for (size_t i = 0; i < TotalLUT.size(); ++i) {
auto& LUT = TotalLUT[i];
constexpr auto GetLUT = [](size_t i, size_t Index) {
if (i & (1U << Index)) {
return -1U;
}
return 0U;
};
LUT.Val[0] = GetLUT(i, 0);
LUT.Val[1] = GetLUT(i, 1);
LUT.Val[2] = GetLUT(i, 2);
LUT.Val[3] = GetLUT(i, 3);
}
return TotalLUT;
}()};
constexpr static auto DPPD_MASK {[]() consteval {
struct LUTType {
uint64_t Val[2];
};
std::array<LUTType, 4> TotalLUT {};
for (size_t i = 0; i < TotalLUT.size(); ++i) {
auto& LUT = TotalLUT[i];
constexpr auto GetLUT = [](size_t i, size_t Index) {
if (i & (1U << Index)) {
return -1ULL;
}
return 0ULL;
};
LUT.Val[0] = GetLUT(i, 0);
LUT.Val[1] = GetLUT(i, 1);
}
return TotalLUT;
}()};
constexpr static auto PBLENDW_LUT {[]() consteval {
struct LUTType {
uint16_t Val[8];
};
// 16-bit words in [127:112], [111:96], [95:80], [79:64], [63:48], [47:32], [31:16], [15:0] are selected using 8-bit swizzle.
// Expectation for this LUT is to simulate PBLENDW with ARM's TBX (one register) instruction.
// PBLENDW behaviour:
// 16-bit words from the source is moved in to the destination based on the bit in the swizzle.
// Dest[15:0] = Swizzle[0] ? Src[15:0] : Dest[15:0]
// Dest[31:16] = Swizzle[1] ? Src[31:16] : Dest[31:16]
// Dest[47:32] = Swizzle[2] ? Src[47:32] : Dest[47:32]
// Dest[63:48] = Swizzle[3] ? Src[63:48] : Dest[63:48]
// Dest[79:64] = Swizzle[4] ? Src[79:64] : Dest[79:64]
// Dest[95:80] = Swizzle[5] ? Src[95:80] : Dest[95:80]
// Dest[111:96] = Swizzle[6] ? Src[111:96] : Dest[111:96]
// Dest[127:112] = Swizzle[7] ? Src[127:112] : Dest[127:112]
std::array<LUTType, 256> TotalLUT {};
const uint16_t WordSelectionSrc[8] = {
0x01'00, 0x03'02, 0x05'04, 0x07'06, 0x09'08, 0x0B'0A, 0x0D'0C, 0x0F'0E,
};
constexpr uint16_t OriginalDest = 0xFF'FF;
for (size_t i = 0; i < 256; ++i) {
auto& LUT = TotalLUT[i];
for (size_t j = 0; j < 8; ++j) {
LUT.Val[j] = ((i >> j) & 1) ? WordSelectionSrc[j] : OriginalDest;
}
}
return TotalLUT;
}()};
CPUBackend::CPUBackend(FEXCore::Core::InternalThreadState* ThreadState, size_t InitialCodeSize, size_t MaxCodeSize)
: ThreadState(ThreadState)
, InitialCodeSize(InitialCodeSize)
, MaxCodeSize(MaxCodeSize) {
auto& Common = ThreadState->CurrentFrame->Pointers.Common;
// Initialize named vector constants.
for (size_t i = 0; i < FEXCore::IR::NamedVectorConstant::NAMED_VECTOR_CONST_POOL_MAX; ++i) {
Common.NamedVectorConstantPointers[i] = reinterpret_cast<uint64_t>(NamedVectorConstants[i]);
}
// Copy named vector constants.
memcpy(Common.NamedVectorConstants, NamedVectorConstants, sizeof(NamedVectorConstants));
// Initialize Indexed named vector constants.
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_PSHUFLW] =
reinterpret_cast<uint64_t>(PSHUFLW_LUT.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_PSHUFHW] =
reinterpret_cast<uint64_t>(PSHUFHW_LUT.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_PSHUFD] =
reinterpret_cast<uint64_t>(PSHUFD_LUT.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_SHUFPS] =
reinterpret_cast<uint64_t>(SHUFPS_LUT.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_DPPS_MASK] =
reinterpret_cast<uint64_t>(DPPS_MASK.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_DPPD_MASK] =
reinterpret_cast<uint64_t>(DPPD_MASK.data());
Common.IndexedNamedVectorConstantPointers[FEXCore::IR::IndexNamedVectorConstant::INDEXED_NAMED_VECTOR_PBLENDW] =
reinterpret_cast<uint64_t>(PBLENDW_LUT.data());
#ifndef FEX_DISABLE_TELEMETRY
// Fill in telemetry values
for (size_t i = 0; i < FEXCore::Telemetry::TYPE_LAST; ++i) {
auto &Telem = FEXCore::Telemetry::GetTelemetryValue(static_cast<FEXCore::Telemetry::TelemetryType>(i));
Common.TelemetryValueAddresses[i] = reinterpret_cast<uint64_t>(Telem.GetAddr());
}
// Fill in telemetry values
for (size_t i = 0; i < FEXCore::Telemetry::TYPE_LAST; ++i) {
auto& Telem = FEXCore::Telemetry::GetTelemetryValue(static_cast<FEXCore::Telemetry::TelemetryType>(i));
Common.TelemetryValueAddresses[i] = reinterpret_cast<uint64_t>(Telem.GetAddr());
}
#endif
}
CPUBackend::~CPUBackend() {
for (auto CodeBuffer : CodeBuffers) {
FreeCodeBuffer(CodeBuffer);
}
CodeBuffers.clear();
}
auto CPUBackend::GetEmptyCodeBuffer() -> CodeBuffer * {
if (ThreadState->CurrentFrame->SignalHandlerRefCounter == 0) {
if (CodeBuffers.empty()) {
CPUBackend::~CPUBackend() {
for (auto CodeBuffer : CodeBuffers) {
FreeCodeBuffer(CodeBuffer);
}
CodeBuffers.clear();
}
auto CPUBackend::GetEmptyCodeBuffer() -> CodeBuffer* {
if (ThreadState->CurrentFrame->SignalHandlerRefCounter == 0) {
if (CodeBuffers.empty()) {
auto NewCodeBuffer = AllocateNewCodeBuffer(InitialCodeSize);
EmplaceNewCodeBuffer(NewCodeBuffer);
} else {
if (CodeBuffers.size() > 1) {
// If we have more than one code buffer we are tracking then walk them and delete
// This is a cleanup step
for (size_t i = 1; i < CodeBuffers.size(); i++) {
FreeCodeBuffer(CodeBuffers[i]);
}
CodeBuffers.resize(1);
}
// Set the current code buffer to the initial
CurrentCodeBuffer = &CodeBuffers[0];
if (CurrentCodeBuffer->Size != MaxCodeSize) {
FreeCodeBuffer(*CurrentCodeBuffer);
// Resize the code buffer and reallocate our code size
CurrentCodeBuffer->Size *= 1.5;
CurrentCodeBuffer->Size = std::min(CurrentCodeBuffer->Size, MaxCodeSize);
*CurrentCodeBuffer = AllocateNewCodeBuffer(CurrentCodeBuffer->Size);
}
}
} else {
// We have signal handlers that have generated code
// This means that we can not safely clear the code at this point in time
// Allocate some new code buffers that we can switch over to instead
auto NewCodeBuffer = AllocateNewCodeBuffer(InitialCodeSize);
EmplaceNewCodeBuffer(NewCodeBuffer);
} else {
if (CodeBuffers.size() > 1) {
// If we have more than one code buffer we are tracking then walk them and delete
// This is a cleanup step
for (size_t i = 1; i < CodeBuffers.size(); i++) {
FreeCodeBuffer(CodeBuffers[i]);
}
CodeBuffers.resize(1);
}
// Set the current code buffer to the initial
CurrentCodeBuffer = &CodeBuffers[0];
}
if (CurrentCodeBuffer->Size != MaxCodeSize) {
FreeCodeBuffer(*CurrentCodeBuffer);
return CurrentCodeBuffer;
}
// Resize the code buffer and reallocate our code size
CurrentCodeBuffer->Size *= 1.5;
CurrentCodeBuffer->Size = std::min(CurrentCodeBuffer->Size, MaxCodeSize);
auto CPUBackend::AllocateNewCodeBuffer(size_t Size) -> CodeBuffer {
#ifndef _WIN32
// MDWE (Memory-Deny-Write-Execute) is a new Linux 6.3 feature.
// It's equivalent to systemd's `MemoryDenyWriteExecute` but implemented entirely in the kernel.
//
// MDWE prevents applications from creating RWX memory mappings.
// This prevents FEX from doing anything JIT related, as FEX uses RWX for JIT memory mappings.
//
// A potential workaround to make FEX work with MDWE is to call mprotect every time we need to write or modify code.
// Alternatively, FEX could use a memory mirror where one half is mapped as RW and the other is RX.
//
// Once MDWE is enabled with the prctl, the feature is sealed and it can /NOT/ be turned off.
//
// Status of MDWE is queried through prctl using `PR_GET_MDWE`:
// -1: The kernel doesn't support MDWE
// 0: MDWE is supported but disabled
// >0: MDWE is enabled, hence prohibiting RWX mappings
#ifndef PR_GET_MDWE
#define PR_GET_MDWE 66
#endif
int MDWE = ::prctl(PR_GET_MDWE, 0, 0, 0, 0);
if (MDWE != -1 && MDWE != 0) {
LogMan::Msg::EFmt("MDWE was set to 0x{:x} which means FEX can't allocate executable memory", MDWE);
}
#endif
*CurrentCodeBuffer = AllocateNewCodeBuffer(CurrentCodeBuffer->Size);
CodeBuffer Buffer;
Buffer.Size = Size;
Buffer.Ptr = static_cast<uint8_t*>(FEXCore::Allocator::VirtualAlloc(Buffer.Size, true));
LOGMAN_THROW_AA_FMT(!!Buffer.Ptr, "Couldn't allocate code buffer");
if (static_cast<Context::ContextImpl*>(ThreadState->CTX)->Config.GlobalJITNaming()) {
static_cast<Context::ContextImpl*>(ThreadState->CTX)->Symbols.RegisterJITSpace(Buffer.Ptr, Buffer.Size);
}
return Buffer;
}
void CPUBackend::FreeCodeBuffer(CodeBuffer Buffer) {
FEXCore::Allocator::VirtualFree(Buffer.Ptr, Buffer.Size);
}
bool CPUBackend::IsAddressInCodeBuffer(uintptr_t Address) const {
for (auto& Buffer : CodeBuffers) {
auto start = (uintptr_t)Buffer.Ptr;
auto end = start + Buffer.Size;
if (Address >= start && Address < end) {
return true;
}
}
} else {
// We have signal handlers that have generated code
// This means that we can not safely clear the code at this point in time
// Allocate some new code buffers that we can switch over to instead
auto NewCodeBuffer = AllocateNewCodeBuffer(InitialCodeSize);
EmplaceNewCodeBuffer(NewCodeBuffer);
return false;
}
return CurrentCodeBuffer;
}
auto CPUBackend::AllocateNewCodeBuffer(size_t Size) -> CodeBuffer {
CodeBuffer Buffer;
Buffer.Size = Size;
Buffer.Ptr = static_cast<uint8_t *>(
FEXCore::Allocator::VirtualAlloc(Buffer.Size, true));
LOGMAN_THROW_AA_FMT(!!Buffer.Ptr, "Couldn't allocate code buffer");
if (static_cast<Context::ContextImpl*>(ThreadState->CTX)->Config.GlobalJITNaming()) {
static_cast<Context::ContextImpl*>(ThreadState->CTX)->Symbols.RegisterJITSpace(Buffer.Ptr, Buffer.Size);
}
return Buffer;
}
void CPUBackend::FreeCodeBuffer(CodeBuffer Buffer) {
FEXCore::Allocator::VirtualFree(Buffer.Ptr, Buffer.Size);
}
bool CPUBackend::IsAddressInCodeBuffer(uintptr_t Address) const {
for (auto &Buffer: CodeBuffers) {
auto start = (uintptr_t)Buffer.Ptr;
auto end = start + Buffer.Size;
if (Address >= start && Address < end) {
return true;
}
}
return false;
}
}
}
} // namespace CPU
} // namespace FEXCore
@@ -20,14 +20,14 @@ namespace FEXCore {
namespace IR {
class IRListView;
class RegisterAllocationData;
}
} // namespace IR
namespace Core {
struct DebugData;
struct ThreadState;
struct CpuStateFrame;
struct InternalThreadState;
}
} // namespace Core
namespace CodeSerialize {
struct CodeObjectFileSection;
@@ -35,30 +35,29 @@ namespace CodeSerialize {
namespace CPU {
struct CPUBackendFeatures {
bool SupportsStaticRegisterAllocation = false;
bool SupportsFlags = false;
bool SupportsSaturatingRoundingShifts = false;
bool SupportsVTBL2 = false;
};
class CPUBackend {
public:
struct CodeBuffer {
uint8_t *Ptr;
uint8_t* Ptr;
size_t Size;
};
/**
* @param InitialCodeSize - Initial size for the code buffers
* @param MaxCodeSize - Max size for the code buffers
*/
CPUBackend(FEXCore::Core::InternalThreadState *ThreadState, size_t InitialCodeSize, size_t MaxCodeSize);
*/
CPUBackend(FEXCore::Core::InternalThreadState* ThreadState, size_t InitialCodeSize, size_t MaxCodeSize);
virtual ~CPUBackend();
/**
* @return The name of this backend
*/
[[nodiscard]] virtual fextl::string GetName() = 0;
[[nodiscard]]
virtual fextl::string GetName() = 0;
struct CompiledCode {
// Where this code block begins.
@@ -99,6 +98,11 @@ namespace CPU {
// Offset after this block to the start of the RIP entries.
uint32_t OffsetToRIPEntries;
// Shared-code modification spin-loop futex.
uint32_t SpinLockFutex;
uint32_t _Pad;
};
// Entries that live after the JITCodeTail.
@@ -133,10 +137,9 @@ namespace CPU {
*
* @return Information about the compiled code block.
*/
[[nodiscard]] virtual CompiledCode CompileCode(uint64_t Entry,
FEXCore::IR::IRListView const *IR,
FEXCore::Core::DebugData *DebugData,
FEXCore::IR::RegisterAllocationData *RAData, bool GDBEnabled) = 0;
[[nodiscard]]
virtual CompiledCode CompileCode(uint64_t Entry, const FEXCore::IR::IRListView* IR, FEXCore::Core::DebugData* DebugData,
const FEXCore::IR::RegisterAllocationData* RAData) = 0;
/**
* @brief Relocates a block of code from the JIT code object cache
@@ -146,21 +149,18 @@ namespace CPU {
*
* @return An executable function pointer relocated from the cache object
*/
[[nodiscard]] virtual void *RelocateJITObjectCode(uint64_t Entry, CodeSerialize::CodeObjectFileSection const *SerializationData) { return nullptr; }
[[nodiscard]]
virtual void* RelocateJITObjectCode(uint64_t Entry, const CodeSerialize::CodeObjectFileSection* SerializationData) {
return nullptr;
}
/**
* @brief Function for mapping memory in to the CPUBackend's visible space. Allows setting up virtual mappings if required
*
* @return Currently unused
*/
[[nodiscard]] virtual void *MapRegion(void *HostPtr, uint64_t GuestPtr, uint64_t Size) = 0;
/**
* @brief This is post-setup initialization that is called just before code executino
*
* Guest memory is available at this point and ThreadState is valid
*/
virtual void Initialize() {}
[[nodiscard]]
virtual void* MapRegion(void* HostPtr, uint64_t GuestPtr, uint64_t Size) = 0;
/**
* @brief Lets FEXCore know if this CPUBackend needs IR and DebugData for CompileCode
@@ -171,7 +171,8 @@ namespace CPU {
*
* @return true if it needs the IR
*/
[[nodiscard]] virtual bool NeedsOpDispatch() = 0;
[[nodiscard]]
virtual bool NeedsOpDispatch() = 0;
virtual void ClearCache() {}
@@ -187,13 +188,14 @@ namespace CPU {
// to be able to handle a 256-bit vector store to a slot.
constexpr static uint32_t MaxSpillSlotSize = 32;
FEXCore::Core::InternalThreadState *ThreadState;
FEXCore::Core::InternalThreadState* ThreadState;
size_t InitialCodeSize, MaxCodeSize;
[[nodiscard]] CodeBuffer *GetEmptyCodeBuffer();
[[nodiscard]]
CodeBuffer* GetEmptyCodeBuffer();
// This is the current code buffer that we are tracking
CodeBuffer *CurrentCodeBuffer{};
CodeBuffer* CurrentCodeBuffer {};
private:
CodeBuffer AllocateNewCodeBuffer(size_t Size);
@@ -205,8 +207,8 @@ namespace CPU {
// This is the array of code buffers. Unless signals force us to keep more than
// buffer, there will be only one entry here
fextl::vector<CodeBuffer> CodeBuffers{};
fextl::vector<CodeBuffer> CodeBuffers {};
};
}
}
} // namespace CPU
} // namespace FEXCore
File diff suppressed because it is too large. Load diff
+94 -82
View File
@@ -14,6 +14,8 @@ namespace Context {
class ContextImpl;
}
uint32_t GetCycleCounterFrequency();
// Debugging define to switch what family of CPU we execute as.
// Might be useful if an application makes an assumption about a CPU.
// #define CPUID_AMD
@@ -28,12 +30,12 @@ private:
constexpr static uint32_t CPUID_VENDOR_AMD3 = 0x444D4163; // "cAMD"
public:
CPUIDEmu(const FEXCore::Context::ContextImpl* ctx);
// X86 cacheline size effectively has to be hardcoded to 64
// if we report anything differently then applications are likely to break
constexpr static uint64_t CACHELINE_SIZE = 64;
void Init(FEXCore::Context::ContextImpl *ctx);
FEXCore::CPUID::FunctionResults RunFunction(uint32_t Function, uint32_t Leaf) const {
if (Function < Primary.size()) {
const auto Handler = Primary[Function];
@@ -56,12 +58,13 @@ public:
}
FEXCore::CPUID::FunctionResults RunFunctionName(uint32_t Function, uint32_t Leaf, uint32_t CPU) const {
if (Function == 0x8000'0002U)
if (Function == 0x8000'0002U) {
return Function_8000_0002h(Leaf, CPU % PerCPUData.size());
else if (Function == 0x8000'0003U)
} else if (Function == 0x8000'0003U) {
return Function_8000_0003h(Leaf, CPU % PerCPUData.size());
else
} else {
return Function_8000_0004h(Leaf, CPU % PerCPUData.size());
}
}
FEXCore::CPUID::XCRResults RunXCRFunction(uint32_t Function) const {
@@ -111,10 +114,12 @@ public:
}
private:
FEXCore::Context::ContextImpl *CTX;
bool Hybrid{};
FEX_CONFIG_OPT(Cores, THREADS);
const FEXCore::Context::ContextImpl* CTX;
bool Hybrid {};
uint32_t Cores {};
FEX_CONFIG_OPT(HideHypervisorBit, HIDEHYPERVISORBIT);
FEX_CONFIG_OPT(SmallTSCScale, SMALLTSCSCALE);
FEX_CONFIG_OPT(MemcpySetTSOEnabled, MEMCPYSETTSOENABLED);
// XFEATURE_ENABLED_MASK
// Mask that configures what features are enabled on the CPU.
@@ -136,11 +141,17 @@ private:
constexpr static uint64_t XCR0_SSE = 1ULL << 1;
constexpr static uint64_t XCR0_AVX = 1ULL << 2;
uint64_t XCR0 {
XCR0_X87 |
XCR0_SSE
struct FeaturesConfig {
uint64_t SHA : 1;
uint64_t _pad : 63;
};
FeaturesConfig Features {
.SHA = 1,
};
uint64_t XCR0 {XCR0_X87 | XCR0_SSE};
uint32_t SupportsAVX() const {
return (XCR0 & XCR0_AVX) ? 1 : 0;
}
@@ -148,13 +159,13 @@ private:
using FunctionHandler = FEXCore::CPUID::FunctionResults (CPUIDEmu::*)(uint32_t Leaf) const;
struct CPUData {
const char *ProductName{};
const char* ProductName {};
#ifdef _M_ARM_64
uint32_t MIDR{};
uint32_t MIDR {};
#endif
bool IsBig{};
bool IsBig {};
};
fextl::vector<CPUData> PerCPUData{};
fextl::vector<CPUData> PerCPUData {};
// Functions
FEXCore::CPUID::FunctionResults Function_0h(uint32_t Leaf) const;
@@ -189,6 +200,7 @@ private:
FEXCore::CPUID::XCRResults XCRFunction_0h() const;
void SetupHostHybridFlag();
void SetupFeatures();
static constexpr size_t PRIMARY_FUNCTION_COUNT = 27;
static constexpr size_t HYPERVISOR_FUNCTION_COUNT = 2;
static constexpr size_t EXTENDED_FUNCTION_COUNT = 32;
@@ -264,74 +276,74 @@ private:
static constexpr std::array<FunctionConstant, PRIMARY_FUNCTION_COUNT> Primary_Constant = {{
// 0: Highest function parameter and ID
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 1: Processor info
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 2: Cache and TLB info
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 3: Serial Number(previously), now reserved
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#ifndef CPUID_AMD
// 4: Deterministic cache parameters for each level
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT},
#else
// 4: Reserved
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#endif
// 5: Monitor/mwait
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 6: Thermal and power management
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 7: Extended feature flags
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT},
// 0x08: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 9: Direct Cache Access information
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x0A: Architectural performance monitoring
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x0B: Extended topology enumeration
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x0C: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x0D: Processor extended state enumeration
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT},
// 0x0E: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x0F: Intel RDT monitoring
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x10: Intel RDT allocation enumeration
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x12: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x12: Intel SGX capability enumeration
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x13: Reserved
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x14: Intel Processor trace
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#ifndef CPUID_AMD
// 0x15: Timestamp counter information
// Doesn't exist on AMD hardware
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#else
// 0x15: Reserved
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#endif
// 0x16: Processor frequency information
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x17: SoC vendor attribute enumeration
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x18: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x19: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#ifndef CPUID_AMD
// 0x1A: Hybrid Information Sub-leaf
{ SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#else
// 0x1A: Reserved
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#endif
}};
@@ -344,9 +356,9 @@ private:
static constexpr std::array<FunctionConstant, HYPERVISOR_FUNCTION_COUNT> Hypervisor_Constant = {{
// Hypervisor CPUID information leaf
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// FEX-Emu specific leaf
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT},
}};
static constexpr std::array<FunctionHandler, EXTENDED_FUNCTION_COUNT> Extended = {
@@ -426,79 +438,79 @@ private:
static constexpr std::array<FunctionConstant, EXTENDED_FUNCTION_COUNT> Extended_Constant = {{
// Largest extended function number
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// Processor vendor
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// Processor brand string
{ SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// Processor brand string continued
{ SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// Processor brand string continued
{ SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::NONCONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#ifdef CPUID_AMD
// 0x8000'0005: L1 Cache and TLB identifiers
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#else
// 0x8000'0005: Reserved
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#endif
// 0x8000'0006: L2 Cache identifiers
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0007: Advanced power management information
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0008: Virtual and physical address sizes
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0009: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'000A: SVM Revision
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'000B: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'000C: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'000D: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'000E: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'000F: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0010: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0011: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0012: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0013: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0014: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0015: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0016: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0017: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0018: Reserved?
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'0019: TLB 1GB page identifiers
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'001A: Performance optimization identifiers
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'001B: Instruction based sampling identifiers
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'001C: Lightweight profiling capabilities
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#ifdef CPUID_AMD
// 0x8000'001D: Cache properties
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NEEDSLEAFCONSTANT},
#else
// 0x8000'001D: Reserved
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
#endif
// 0x8000'001E: Extended APIC ID
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
// 0x8000'001F: AMD Secure Encryption
{ SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT },
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
}};
};
}
} // namespace FEXCore
File diff suppressed because it is too large. Load diff
@@ -1,6 +0,0 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <stdint.h>
namespace FEXCore::CPU {
}
@@ -1,20 +1,23 @@
// SPDX-License-Identifier: MIT
#include "Interface/Context/Context.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include "Interface/Core/LookupCache.h"
#include "Interface/Core/X86HelperGen.h"
#include "Utils/MemberFunctionToPointer.h"
#include <FEXCore/Config/Config.h>
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Core/SignalDelegator.h>
#include <FEXCore/Core/X86Enums.h>
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXCore/HLE/SyscallHandler.h>
#include <FEXCore/Utils/Event.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/MathUtils.h>
#include <CodeEmitter/Emitter.h>
#include <atomic>
#include <condition_variable>
#include <csignal>
@@ -23,45 +26,25 @@
namespace FEXCore::CPU {
void Dispatcher::SleepThread(FEXCore::Context::ContextImpl *ctx, FEXCore::Core::CpuStateFrame *Frame) {
auto Thread = Frame->Thread;
--ctx->IdleWaitRefCount;
ctx->IdleWaitCV.notify_all();
Thread->RunningEvents.ThreadSleeping = true;
// Go to sleep
Thread->StartRunning.Wait();
Thread->RunningEvents.Running = true;
++ctx->IdleWaitRefCount;
Thread->RunningEvents.ThreadSleeping = false;
ctx->IdleWaitCV.notify_all();
static void SleepThread(FEXCore::Context::ContextImpl* CTX, FEXCore::Core::CpuStateFrame* Frame) {
CTX->SyscallHandler->SleepThread(CTX, Frame);
}
uint64_t Dispatcher::GetCompileBlockPtr() {
using ClassPtrType = void (FEXCore::Context::ContextImpl::*)(FEXCore::Core::CpuStateFrame *, uint64_t);
union PtrCast {
ClassPtrType ClassPtr;
uintptr_t Data;
};
constexpr size_t MAX_DISPATCHER_CODE_SIZE = 4096 * 2;
PtrCast CompileBlockPtr;
CompileBlockPtr.ClassPtr = &FEXCore::Context::ContextImpl::CompileBlockJit;
return CompileBlockPtr.Data;
}
constexpr size_t MAX_DISPATCHER_CODE_SIZE = 4096;
Dispatcher::Dispatcher(FEXCore::Context::ContextImpl *ctx, const DispatcherConfig &config)
: Arm64Emitter(ctx, MAX_DISPATCHER_CODE_SIZE)
, CTX {ctx}
, config {config} {
Dispatcher::Dispatcher(FEXCore::Context::ContextImpl* ctx)
: Arm64Emitter(ctx, FEXCore::Allocator::VirtualAlloc(MAX_DISPATCHER_CODE_SIZE, true), MAX_DISPATCHER_CODE_SIZE)
, CTX {ctx} {
EmitDispatcher();
}
Dispatcher::~Dispatcher() {
auto BufferSize = GetBufferSize();
if (BufferSize) {
FEXCore::Allocator::VirtualFree(GetBufferBase(), BufferSize);
}
}
void Dispatcher::EmitDispatcher() {
#ifdef VIXL_DISASSEMBLER
const auto DisasmBegin = GetCursorAddress<const vixl::aarch64::Instruction*>();
@@ -78,8 +61,16 @@ void Dispatcher::EmitDispatcher() {
// }
ARMEmitter::ForwardLabel l_CTX;
ARMEmitter::ForwardLabel l_Sleep;
ARMEmitter::ForwardLabel l_CompileBlock;
ARMEmitter::SingleUseForwardLabel l_Sleep;
#ifdef _M_ARM_64EC
// These structures are not included in the standard Windows headers, define them here
static constexpr size_t TEBCPUAreaOffset = 0x1788;
static constexpr size_t CPUAreaInSyscallCallbackOffset = 0x1;
static constexpr size_t CPUAreaEmulatorStackLimitOffset = 0x8;
static constexpr size_t CPUAreaEmulatorDataOffset = 0x30;
ARMEmitter::SingleUseForwardLabel ExitEC;
#endif
ARMEmitter::SingleUseForwardLabel l_CompileBlock;
// Push all the register we need to save
PushCalleeSavedRegisters();
@@ -96,101 +87,137 @@ void Dispatcher::EmitDispatcher() {
AbsoluteLoopTopAddressFillSRA = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation) {
FillStaticRegs();
FillStaticRegs();
ARMEmitter::BiDirectionalLabel LoopTop {};
#ifdef _M_ARM_64EC
b(&LoopTop);
AbsoluteLoopTopAddressEnterECFillSRA = GetCursorAddress<uint64_t>();
ldr(STATE, EC_ENTRY_CPUAREA_REG, CPUAreaEmulatorDataOffset);
FillStaticRegs();
// Enter JIT
b(&LoopTop);
AbsoluteLoopTopAddressEnterEC = GetCursorAddress<uint64_t>();
// Load ThreadState and write the target PC there
ldr(STATE, EC_ENTRY_CPUAREA_REG, CPUAreaEmulatorDataOffset);
str(EC_CALL_CHECKER_PC_REG, STATE_PTR(CpuStateFrame, State.rip));
// Swap stacks to the emulator stack
ldr(TMP1, EC_ENTRY_CPUAREA_REG, CPUAreaEmulatorStackLimitOffset);
add(ARMEmitter::Size::i64Bit, StaticRegisters[X86State::REG_RSP], ARMEmitter::Reg::rsp, 0);
add(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::rsp, TMP1, 0);
if (EmitterCTX->HostFeatures.SupportsSVE128) {
ptrue(ARMEmitter::SubRegSize::i8Bit, PRED_TMP_16B, ARMEmitter::PredicatePattern::SVE_VL16);
}
// Enter JIT
#endif
// We want to ensure that we are 16 byte aligned at the top of this loop
Align16B();
ARMEmitter::BiDirectionalLabel FullLookup{};
ARMEmitter::BiDirectionalLabel CallBlock{};
ARMEmitter::BackwardLabel LoopTop{};
ARMEmitter::BiDirectionalLabel FullLookup {};
ARMEmitter::BiDirectionalLabel CallBlock {};
Bind(&LoopTop);
AbsoluteLoopTopAddress = GetCursorAddress<uint64_t>();
// Load in our RIP
// Don't modify x2 since it contains our RIP once the block doesn't exist
auto RipReg = ARMEmitter::XReg::x2;
// Don't modify TMP3 since it contains our RIP once the block doesn't exist
auto RipReg = TMP3;
ldr(RipReg, STATE_PTR(CpuStateFrame, State.rip));
// L1 Cache
ldr(ARMEmitter::XReg::x0, STATE_PTR(CpuStateFrame, Pointers.Common.L1Pointer));
ldr(TMP1, STATE_PTR(CpuStateFrame, Pointers.Common.L1Pointer));
and_(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, RipReg.R(), LookupCache::L1_ENTRIES_MASK);
add(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, ARMEmitter::Reg::r0, ARMEmitter::Reg::r3, ARMEmitter::ShiftType::LSL , 4);
ldp<ARMEmitter::IndexType::OFFSET>(ARMEmitter::XReg::x3, ARMEmitter::XReg::x0, ARMEmitter::Reg::r0, 0);
cmp(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, RipReg.R());
b(ARMEmitter::Condition::CC_NE, &FullLookup);
and_(ARMEmitter::Size::i64Bit, TMP4, RipReg.R(), LookupCache::L1_ENTRIES_MASK);
add(ARMEmitter::Size::i64Bit, TMP1, TMP1, TMP4, ARMEmitter::ShiftType::LSL, 4);
ldp<ARMEmitter::IndexType::OFFSET>(TMP4, TMP1, TMP1, 0);
sub(ARMEmitter::Size::i64Bit, TMP1, TMP1, RipReg);
cbnz(ARMEmitter::Size::i64Bit, TMP1, &FullLookup);
br(ARMEmitter::Reg::r3);
br(TMP4);
// L1C check failed, do a full lookup
Bind(&FullLookup);
// This is the block cache lookup routine
// It matches what is going on it LookupCache.h::FindBlock
ldr(ARMEmitter::XReg::x0, STATE_PTR(CpuStateFrame, Pointers.Common.L2Pointer));
ldr(TMP1, STATE_PTR(CpuStateFrame, Pointers.Common.L2Pointer));
// Mask the address by the virtual address size so we can check for aliases
uint64_t VirtualMemorySize = CTX->Config.VirtualMemSize;
if (std::popcount(VirtualMemorySize) == 1) {
and_(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, RipReg.R(), VirtualMemorySize - 1);
}
else {
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, VirtualMemorySize);
and_(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, RipReg.R(), ARMEmitter::Reg::r3);
and_(ARMEmitter::Size::i64Bit, TMP4, RipReg.R(), VirtualMemorySize - 1);
} else {
LoadConstant(ARMEmitter::Size::i64Bit, TMP4, VirtualMemorySize);
and_(ARMEmitter::Size::i64Bit, TMP4, RipReg.R(), TMP4);
}
ARMEmitter::ForwardLabel NoBlock;
{
// Offset the address and add to our page pointer
lsr(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, ARMEmitter::Reg::r3, 12);
lsr(ARMEmitter::Size::i64Bit, TMP2, TMP4, 12);
// Load the pointer from the offset
ldr(ARMEmitter::XReg::x0, ARMEmitter::Reg::r0, ARMEmitter::Reg::r1, ARMEmitter::ExtendedType::LSL_64, 3);
ldr(TMP1, TMP1, TMP2, ARMEmitter::ExtendedType::LSL_64, 3);
// If page pointer is zero then we have no block
cbz(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, &NoBlock);
cbz(ARMEmitter::Size::i64Bit, TMP1, &NoBlock);
#ifdef _M_ARM_64EC
// The LSB of an L2 page entry indicates if this page contains EC code
tbnz(TMP1, 0, &ExitEC);
#endif
// Steal the page offset
and_(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, ARMEmitter::Reg::r3, 0x0FFF);
and_(ARMEmitter::Size::i64Bit, TMP2, TMP4, 0x0FFF);
// Shift the offset by the size of the block cache entry
add(ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, ARMEmitter::XReg::x1, ARMEmitter::ShiftType::LSL, (int)log2(sizeof(FEXCore::LookupCache::LookupCacheEntry)));
add(TMP1, TMP1, TMP2, ARMEmitter::ShiftType::LSL, (int)log2(sizeof(FEXCore::LookupCache::LookupCacheEntry)));
// The the full LookupCacheEntry with a single LDP.
// Check the guest address first to ensure it maps to the address we are currently at.
// This fixes aliasing problems
ldp<ARMEmitter::IndexType::OFFSET>(ARMEmitter::XReg::x3, ARMEmitter::XReg::x1, ARMEmitter::Reg::r0, 0);
ldp<ARMEmitter::IndexType::OFFSET>(TMP4, TMP2, TMP1, 0);
// If the guest address doesn't match, Compile the block.
cmp(ARMEmitter::XReg::x1, RipReg);
b(ARMEmitter::Condition::CC_NE, &NoBlock);
sub(TMP2, TMP2, RipReg);
cbnz(ARMEmitter::Size::i64Bit, TMP2, &NoBlock);
// Check the host address to see if it matches, else compile the block.
cbz(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, &NoBlock);
cbz(ARMEmitter::Size::i64Bit, TMP4, &NoBlock);
// If we've made it here then we have a real compiled block
{
// update L1 cache
ldr(ARMEmitter::XReg::x0, STATE_PTR(CpuStateFrame, Pointers.Common.L1Pointer));
ldr(TMP1, STATE_PTR(CpuStateFrame, Pointers.Common.L1Pointer));
and_(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, RipReg.R(), LookupCache::L1_ENTRIES_MASK);
add(ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, ARMEmitter::XReg::x1, ARMEmitter::ShiftType::LSL, 4);
stp<ARMEmitter::IndexType::OFFSET>(ARMEmitter::XReg::x3, ARMEmitter::XReg::x2, ARMEmitter::Reg::r0);
and_(ARMEmitter::Size::i64Bit, TMP2, RipReg.R(), LookupCache::L1_ENTRIES_MASK);
add(TMP1, TMP1, TMP2, ARMEmitter::ShiftType::LSL, 4);
stp<ARMEmitter::IndexType::OFFSET>(TMP4, TMP3, TMP1);
// Jump to the block
br(ARMEmitter::Reg::r3);
br(TMP4);
}
}
#ifdef _M_ARM_64EC
{
Bind(&ExitEC);
add(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::rsp, StaticRegisters[X86State::REG_RSP], 0);
mov(EC_CALL_CHECKER_PC_REG, RipReg);
ldr(TMP2, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionEC));
br(TMP2);
}
#endif
{
ThreadStopHandlerAddressSpillSRA = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation)
SpillStaticRegs(TMP1);
SpillStaticRegs(TMP1);
ThreadStopHandlerAddress = GetCursorAddress<uint64_t>();
@@ -203,71 +230,96 @@ void Dispatcher::EmitDispatcher() {
{
ExitFunctionLinkerAddress = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation)
SpillStaticRegs(TMP1);
SpillStaticRegs(TMP1);
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
add(ARMEmitter::Size::i64Bit, ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, 1);
str(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
#ifdef _M_ARM_64EC
ldr(ARMEmitter::XReg::x0, ARMEmitter::XReg::x18, TEBCPUAreaOffset);
LoadConstant(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r1, 1);
strb(ARMEmitter::WReg::w1, ARMEmitter::XReg::x0, CPUAreaInSyscallCallbackOffset);
#endif
mov(ARMEmitter::XReg::x0, STATE);
mov(ARMEmitter::XReg::x1, ARMEmitter::XReg::lr);
ldr(ARMEmitter::XReg::x2, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionLink));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uintptr_t, void *, void *>(ARMEmitter::Reg::r2);
}
else {
GenerateIndirectRuntimeCall<uintptr_t, void*, void*>(ARMEmitter::Reg::r2);
} else {
blr(ARMEmitter::Reg::r2);
}
if (config.StaticRegisterAllocation)
FillStaticRegs();
if (!TMP_ABIARGS) {
mov(TMP1, ARMEmitter::XReg::x0);
}
ldr(ARMEmitter::XReg::x1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
subs(ARMEmitter::Size::i64Bit, ARMEmitter::XReg::x1, ARMEmitter::XReg::x1, 1);
str(ARMEmitter::XReg::x1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
FillStaticRegs();
#ifdef _M_ARM_64EC
ldr(TMP2, ARMEmitter::XReg::x18, TEBCPUAreaOffset);
strb(ARMEmitter::WReg::zr, TMP2, CPUAreaInSyscallCallbackOffset);
#endif
ldr(TMP2, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
sub(ARMEmitter::Size::i64Bit, TMP2, TMP2, 1);
str(TMP2, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
// Trigger segfault if any deferred signals are pending
ldr(ARMEmitter::XReg::x1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalFaultAddress));
str(ARMEmitter::XReg::zr, ARMEmitter::XReg::x1, 0);
strb(ARMEmitter::XReg::zr, STATE,
offsetof(FEXCore::Core::InternalThreadState, InterruptFaultPage) - offsetof(FEXCore::Core::InternalThreadState, BaseFrameState));
br(ARMEmitter::Reg::r0);
br(TMP1);
}
// Need to create the block
{
Bind(&NoBlock);
if (config.StaticRegisterAllocation)
SpillStaticRegs(TMP1);
SpillStaticRegs(TMP1);
if (!TMP_ABIARGS) {
mov(ARMEmitter::XReg::x2, TMP3);
}
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
add(ARMEmitter::Size::i64Bit, ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, 1);
str(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
#ifdef _M_ARM_64EC
ldr(ARMEmitter::XReg::x0, ARMEmitter::XReg::x18, TEBCPUAreaOffset);
LoadConstant(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r1, 1);
strb(ARMEmitter::WReg::w1, ARMEmitter::XReg::x0, CPUAreaInSyscallCallbackOffset);
#endif
ldr(ARMEmitter::XReg::x0, &l_CTX);
mov(ARMEmitter::XReg::x1, STATE);
ldr(ARMEmitter::XReg::x3, &l_CompileBlock);
// x2 contains guest RIP
mov(ARMEmitter::XReg::x3, 0);
ldr(ARMEmitter::XReg::x4, &l_CompileBlock);
// X2 contains our guest RIP
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<void, void *, uint64_t, void *>(ARMEmitter::Reg::r3);
}
else {
blr(ARMEmitter::Reg::r3); // { CTX, Frame, RIP}
GenerateIndirectRuntimeCall<uintptr_t, void*, void*, uint64_t, uint64_t>(ARMEmitter::Reg::r4);
} else {
blr(ARMEmitter::Reg::r4); // { CTX, Frame, RIP, MaxInst }
}
if (config.StaticRegisterAllocation)
FillStaticRegs();
FillStaticRegs();
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
subs(ARMEmitter::Size::i64Bit, ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, 1);
str(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
#ifdef _M_ARM_64EC
ldr(TMP1, ARMEmitter::XReg::x18, TEBCPUAreaOffset);
strb(ARMEmitter::WReg::zr, TMP1, CPUAreaInSyscallCallbackOffset);
#endif
ldr(TMP1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
sub(ARMEmitter::Size::i64Bit, TMP1, TMP1, 1);
str(TMP1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
// Trigger segfault if any deferred signals are pending
ldr(TMP1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalFaultAddress));
str(ARMEmitter::XReg::zr, TMP1, 0);
strb(ARMEmitter::XReg::zr, STATE,
offsetof(FEXCore::Core::InternalThreadState, InterruptFaultPage) - offsetof(FEXCore::Core::InternalThreadState, BaseFrameState));
b(&LoopTop);
}
@@ -293,8 +345,7 @@ void Dispatcher::EmitDispatcher() {
// Needs to be distinct from the SignalHandlerReturnAddress
GuestSignal_SIGILL = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation)
SpillStaticRegs(TMP1);
SpillStaticRegs(TMP1);
hlt(0);
}
@@ -302,10 +353,9 @@ void Dispatcher::EmitDispatcher() {
{
// Guest SIGTRAP handler
// Needs to be distinct from the SignalHandlerReturnAddress
GuestSignal_SIGTRAP = GetCursorAddress<uint64_t>();
GuestSignal_SIGTRAP = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation)
SpillStaticRegs(TMP1);
SpillStaticRegs(TMP1);
brk(0);
}
@@ -315,8 +365,7 @@ void Dispatcher::EmitDispatcher() {
// Needs to be distinct from the SignalHandlerReturnAddress
GuestSignal_SIGSEGV = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation)
SpillStaticRegs(TMP1);
SpillStaticRegs(TMP1);
// hlt/udf = SIGILL
// brk = SIGTRAP
@@ -327,8 +376,7 @@ void Dispatcher::EmitDispatcher() {
add(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::rsp, ARMEmitter::Reg::r0, 0);
PopCalleeSavedRegisters();
ret();
}
else {
} else {
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, 0);
ldr(ARMEmitter::XReg::x1, ARMEmitter::Reg::r1);
}
@@ -336,8 +384,7 @@ void Dispatcher::EmitDispatcher() {
{
ThreadPauseHandlerAddressSpillSRA = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation)
SpillStaticRegs(TMP1);
SpillStaticRegs(TMP1);
ThreadPauseHandlerAddress = GetCursorAddress<uint64_t>();
// We are pausing, this means the frontend should be waiting for this thread to idle
@@ -348,9 +395,8 @@ void Dispatcher::EmitDispatcher() {
mov(ARMEmitter::XReg::x1, STATE);
ldr(ARMEmitter::XReg::x2, &l_Sleep);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<void, void *, void *>(ARMEmitter::Reg::r2);
}
else {
GenerateIndirectRuntimeCall<void, void*, void*>(ARMEmitter::Reg::r2);
} else {
blr(ARMEmitter::Reg::r2);
}
@@ -393,7 +439,7 @@ void Dispatcher::EmitDispatcher() {
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, CTX->X86CodeGen.CallbackReturn);
ldr(ARMEmitter::XReg::x2, STATE_PTR(CpuStateFrame, State.gregs[X86State::REG_RSP]));
sub(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r2, ARMEmitter::Reg::r2, 16);
sub(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r2, ARMEmitter::Reg::r2, CTX->Config.Is64BitMode ? 16 : 12);
str(ARMEmitter::XReg::x2, STATE_PTR(CpuStateFrame, State.gregs[X86State::REG_RSP]));
// Store the trampoline to the guest stack
@@ -404,112 +450,58 @@ void Dispatcher::EmitDispatcher() {
str(ARMEmitter::XReg::x1, STATE_PTR(CpuStateFrame, State.rip));
// load static regs
if (config.StaticRegisterAllocation)
FillStaticRegs();
FillStaticRegs();
// Now go back to the regular dispatcher loop
b(&LoopTop);
}
{
LUDIVHandlerAddress = GetCursorAddress<uint64_t>();
auto EmitLongALUOpHandler = [&](auto R, auto Offset) {
auto Address = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR(ARMEmitter::Reg::r3);
SpillStaticRegs(ARMEmitter::Reg::r3);
PushDynamicRegsAndLR(TMP4);
SpillStaticRegs(TMP4);
ldr(ARMEmitter::XReg::x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LUDIV));
if (!TMP_ABIARGS) {
mov(ARMEmitter::XReg::x0, TMP1);
mov(ARMEmitter::XReg::x1, TMP2);
mov(ARMEmitter::XReg::x2, TMP3);
}
ldr(ARMEmitter::XReg::x3, R, Offset);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(ARMEmitter::Reg::r3);
}
else {
} else {
blr(ARMEmitter::Reg::r3);
}
// Result is now in x0
if (!TMP_ABIARGS) {
mov(TMP1, ARMEmitter::XReg::x0);
}
FillStaticRegs();
// Result is now in x0
// Fix the stack and any values that were stepped on
PopDynamicRegsAndLR();
// Go back to our code block
ret();
}
return Address;
};
{
LDIVHandlerAddress = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR(ARMEmitter::Reg::r3);
SpillStaticRegs(ARMEmitter::Reg::r3);
ldr(ARMEmitter::XReg::x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LDIV));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(ARMEmitter::Reg::r3);
}
else {
blr(ARMEmitter::Reg::r3);
}
FillStaticRegs();
// Result is now in x0
// Fix the stack and any values that were stepped on
PopDynamicRegsAndLR();
// Go back to our code block
ret();
}
{
LUREMHandlerAddress = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR(ARMEmitter::Reg::r3);
SpillStaticRegs(ARMEmitter::Reg::r3);
ldr(ARMEmitter::XReg::x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LUREM));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(ARMEmitter::Reg::r3);
}
else {
blr(ARMEmitter::Reg::r3);
}
FillStaticRegs();
// Result is now in x0
// Fix the stack and any values that were stepped on
PopDynamicRegsAndLR();
// Go back to our code block
ret();
}
{
LREMHandlerAddress = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR(ARMEmitter::Reg::r3);
SpillStaticRegs(ARMEmitter::Reg::r3);
ldr(ARMEmitter::XReg::x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LREM));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(ARMEmitter::Reg::r3);
}
else {
blr(ARMEmitter::Reg::r3);
}
FillStaticRegs();
// Result is now in x0
// Fix the stack and any values that were stepped on
PopDynamicRegsAndLR();
// Go back to our code block
ret();
}
LUDIVHandlerAddress = EmitLongALUOpHandler(STATE_PTR(CpuStateFrame, Pointers.AArch64.LUDIV));
LDIVHandlerAddress = EmitLongALUOpHandler(STATE_PTR(CpuStateFrame, Pointers.AArch64.LDIV));
LUREMHandlerAddress = EmitLongALUOpHandler(STATE_PTR(CpuStateFrame, Pointers.AArch64.LUREM));
LREMHandlerAddress = EmitLongALUOpHandler(STATE_PTR(CpuStateFrame, Pointers.AArch64.LREM));
Bind(&l_CTX);
dc64(reinterpret_cast<uintptr_t>(CTX));
Bind(&l_Sleep);
dc64(reinterpret_cast<uint64_t>(SleepThread));
Bind(&l_CompileBlock);
dc64(GetCompileBlockPtr());
FEXCore::Utils::MemberFunctionToPointerCast PMF(&FEXCore::Context::ContextImpl::CompileBlock);
dc64(PMF.GetConvertedPointer());
Start = reinterpret_cast<uint64_t>(DispatchPtr);
End = GetCursorAddress<uint64_t>();
@@ -528,7 +520,7 @@ void Dispatcher::EmitDispatcher() {
const auto DisasmEnd = GetCursorAddress<const vixl::aarch64::Instruction*>();
for (auto PCToDecode = DisasmBegin; PCToDecode < DisasmEnd; PCToDecode += 4) {
DisasmDecoder->Decode(PCToDecode);
auto Output = Disasm.GetOutput();
auto Output = Disasm->GetOutput();
LogMan::Msg::IFmt("{}", Output);
}
}
@@ -536,60 +528,28 @@ void Dispatcher::EmitDispatcher() {
}
#ifdef VIXL_SIMULATOR
void Dispatcher::ExecuteDispatch(FEXCore::Core::CpuStateFrame *Frame) {
void Dispatcher::ExecuteDispatch(FEXCore::Core::CpuStateFrame* Frame) {
Simulator.WriteXRegister(0, reinterpret_cast<int64_t>(Frame));
Simulator.RunFrom(reinterpret_cast<vixl::aarch64::Instruction const*>(DispatchPtr));
Simulator.RunFrom(reinterpret_cast< const vixl::aarch64::Instruction*>(DispatchPtr));
}
void Dispatcher::ExecuteJITCallback(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP) {
void Dispatcher::ExecuteJITCallback(FEXCore::Core::CpuStateFrame* Frame, uint64_t RIP) {
Simulator.WriteXRegister(0, reinterpret_cast<int64_t>(Frame));
Simulator.WriteXRegister(1, RIP);
Simulator.RunFrom(reinterpret_cast<vixl::aarch64::Instruction const*>(CallbackPtr));
Simulator.RunFrom(reinterpret_cast< const vixl::aarch64::Instruction*>(CallbackPtr));
}
#endif
size_t Dispatcher::GenerateGDBPauseCheck(uint8_t *CodeBuffer, uint64_t GuestRIP) {
FEXCore::ARMEmitter::Emitter emit{CodeBuffer, MaxGDBPauseCheckSize};
ARMEmitter::ForwardLabel RunBlock;
// If we have a gdb server running then run in a less efficient mode that checks if we need to exit
// This happens when single stepping
static_assert(sizeof(FEXCore::Context::ContextImpl::Config.RunningMode) == 4, "This is expected to be size of 4");
emit.ldr(ARMEmitter::XReg::x0, STATE_PTR(CpuStateFrame, Thread));
emit.ldr(ARMEmitter::XReg::x0, ARMEmitter::Reg::r0, offsetof(FEXCore::Core::InternalThreadState, CTX)); // Get Context
emit.ldr(ARMEmitter::WReg::w0, ARMEmitter::Reg::r0, offsetof(FEXCore::Context::ContextImpl, Config.RunningMode));
// If the value == 0 then we don't need to stop
emit.cbz(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r0, &RunBlock);
{
ARMEmitter::ForwardLabel l_GuestRIP;
// Make sure RIP is syncronized to the context
emit.ldr(ARMEmitter::XReg::x0, &l_GuestRIP);
emit.str(ARMEmitter::XReg::x0, STATE_PTR(CpuStateFrame, State.rip));
// Stop the thread
emit.ldr(ARMEmitter::XReg::x0, STATE_PTR(CpuStateFrame, Pointers.Common.ThreadPauseHandlerSpillSRA));
emit.br(ARMEmitter::Reg::r0);
emit.Bind(&l_GuestRIP);
emit.dc64(GuestRIP);
}
emit.Bind(&RunBlock);
auto UsedBytes = emit.GetCursorOffset();
emit.ClearICache(CodeBuffer, UsedBytes);
return UsedBytes;
}
void Dispatcher::InitThreadPointers(FEXCore::Core::InternalThreadState *Thread) {
void Dispatcher::InitThreadPointers(FEXCore::Core::InternalThreadState* Thread) {
// Setup dispatcher specific pointers that need to be accessed from JIT code
{
auto &Common = Thread->CurrentFrame->Pointers.Common;
auto& Common = Thread->CurrentFrame->Pointers.Common;
Common.DispatcherLoopTop = AbsoluteLoopTopAddress;
Common.DispatcherLoopTopFillSRA = AbsoluteLoopTopAddressFillSRA;
Common.DispatcherLoopTopEnterEC = AbsoluteLoopTopAddressEnterEC;
Common.DispatcherLoopTopEnterECFillSRA = AbsoluteLoopTopAddressEnterECFillSRA;
Common.ExitFunctionLinker = ExitFunctionLinkerAddress;
Common.ThreadStopHandlerSpillSRA = ThreadStopHandlerAddressSpillSRA;
Common.ThreadPauseHandlerSpillSRA = ThreadPauseHandlerAddressSpillSRA;
@@ -599,7 +559,7 @@ void Dispatcher::InitThreadPointers(FEXCore::Core::InternalThreadState *Thread)
Common.SignalReturnHandler = SignalHandlerReturnAddress;
Common.SignalReturnHandlerRT = SignalHandlerReturnAddressRT;
auto &AArch64 = Thread->CurrentFrame->Pointers.AArch64;
auto& AArch64 = Thread->CurrentFrame->Pointers.AArch64;
AArch64.LUDIVHandler = LUDIVHandlerAddress;
AArch64.LDIVHandler = LDIVHandlerAddress;
AArch64.LUREMHandler = LUREMHandlerAddress;
@@ -607,8 +567,8 @@ void Dispatcher::InitThreadPointers(FEXCore::Core::InternalThreadState *Thread)
}
}
fextl::unique_ptr<Dispatcher> Dispatcher::Create(FEXCore::Context::ContextImpl *CTX, const DispatcherConfig &Config) {
return fextl::make_unique<Dispatcher>(CTX, Config);
fextl::unique_ptr<Dispatcher> Dispatcher::Create(FEXCore::Context::ContextImpl* CTX) {
return fextl::make_unique<Dispatcher>(CTX);
}
}
} // namespace FEXCore::CPU
@@ -2,8 +2,8 @@
#pragma once
#include "Interface/Core/ArchHelpers/Arm64Emitter.h"
#include "Interface/Core/CPUBackend.h"
#include <FEXCore/Core/CPUBackend.h>
#include <FEXCore/fextl/memory.h>
#ifdef VIXL_SIMULATOR
@@ -23,7 +23,7 @@ struct GuestSigAction;
namespace FEXCore::Core {
struct CpuStateFrame;
struct InternalThreadState;
}
} // namespace FEXCore::Core
namespace FEXCore::Context {
class ContextImpl;
@@ -31,66 +31,60 @@ class ContextImpl;
namespace FEXCore::CPU {
struct DispatcherConfig {
bool StaticRegisterAllocation = false;
};
#define STATE_PTR(STATE_TYPE, FIELD) \
STATE.R(), offsetof(FEXCore::Core::STATE_TYPE, FIELD)
#define STATE_PTR(STATE_TYPE, FIELD) STATE.R(), offsetof(FEXCore::Core::STATE_TYPE, FIELD)
class Dispatcher final : public Arm64Emitter {
public:
static fextl::unique_ptr<Dispatcher> Create(FEXCore::Context::ContextImpl *CTX, const DispatcherConfig &Config);
static fextl::unique_ptr<Dispatcher> Create(FEXCore::Context::ContextImpl* CTX);
Dispatcher(FEXCore::Context::ContextImpl *ctx, const DispatcherConfig &Config);
~Dispatcher() = default;
Dispatcher(FEXCore::Context::ContextImpl* ctx);
~Dispatcher();
/**
* @name Dispatch Helper functions
* @{ */
uint64_t ThreadStopHandlerAddress{};
uint64_t ThreadStopHandlerAddressSpillSRA{};
uint64_t AbsoluteLoopTopAddress{};
uint64_t AbsoluteLoopTopAddressFillSRA{};
uint64_t ThreadPauseHandlerAddress{};
uint64_t ThreadPauseHandlerAddressSpillSRA{};
uint64_t ExitFunctionLinkerAddress{};
uint64_t SignalHandlerReturnAddress{};
uint64_t SignalHandlerReturnAddressRT{};
uint64_t GuestSignal_SIGILL{};
uint64_t GuestSignal_SIGTRAP{};
uint64_t GuestSignal_SIGSEGV{};
uint64_t IntCallbackReturnAddress{};
uint64_t ThreadStopHandlerAddress {};
uint64_t ThreadStopHandlerAddressSpillSRA {};
uint64_t AbsoluteLoopTopAddress {};
uint64_t AbsoluteLoopTopAddressFillSRA {};
uint64_t AbsoluteLoopTopAddressEnterEC {};
uint64_t AbsoluteLoopTopAddressEnterECFillSRA {};
uint64_t ThreadPauseHandlerAddress {};
uint64_t ThreadPauseHandlerAddressSpillSRA {};
uint64_t ExitFunctionLinkerAddress {};
uint64_t SignalHandlerReturnAddress {};
uint64_t SignalHandlerReturnAddressRT {};
uint64_t GuestSignal_SIGILL {};
uint64_t GuestSignal_SIGTRAP {};
uint64_t GuestSignal_SIGSEGV {};
uint64_t IntCallbackReturnAddress {};
uint64_t PauseReturnInstruction{};
uint64_t PauseReturnInstruction {};
/** @} */
uint64_t Start{};
uint64_t End{};
uint64_t Start {};
uint64_t End {};
void InitThreadPointers(FEXCore::Core::InternalThreadState *Thread);
// These are across all arches for now
static constexpr size_t MaxGDBPauseCheckSize = 128;
size_t GenerateGDBPauseCheck(uint8_t *CodeBuffer, uint64_t GuestRIP);
void InitThreadPointers(FEXCore::Core::InternalThreadState* Thread);
#ifdef VIXL_SIMULATOR
void ExecuteDispatch(FEXCore::Core::CpuStateFrame *Frame) ;
void ExecuteJITCallback(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP);
void ExecuteDispatch(FEXCore::Core::CpuStateFrame* Frame);
void ExecuteJITCallback(FEXCore::Core::CpuStateFrame* Frame, uint64_t RIP);
#else
void ExecuteDispatch(FEXCore::Core::CpuStateFrame *Frame) {
void ExecuteDispatch(FEXCore::Core::CpuStateFrame* Frame) {
DispatchPtr(Frame);
}
void ExecuteJITCallback(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP) {
void ExecuteJITCallback(FEXCore::Core::CpuStateFrame* Frame, uint64_t RIP) {
CallbackPtr(Frame, RIP);
}
#endif
uint16_t GetSRAGPRCount() const {
return StaticRegisters.size();
// PF/AF are the final two SRA registers.
// Only return the SRA for GPRs.
return StaticRegisters.size() - 2;
}
uint16_t GetSRAFPRCount() const {
@@ -98,7 +92,7 @@ public:
}
void GetSRAGPRMapping(uint8_t Mapping[16]) const {
for (size_t i = 0; i < StaticRegisters.size(); ++i) {
for (size_t i = 0; i < StaticRegisters.size() - 2; ++i) {
Mapping[i] = StaticRegisters[i].Idx();
}
}
@@ -109,29 +103,22 @@ public:
}
}
const DispatcherConfig& GetConfig() const { return config; }
protected:
FEXCore::Context::ContextImpl *CTX;
DispatcherConfig config;
FEXCore::Context::ContextImpl* CTX;
static void SleepThread(FEXCore::Context::ContextImpl *ctx, FEXCore::Core::CpuStateFrame *Frame);
static uint64_t GetCompileBlockPtr();
using AsmDispatch = void(*)(FEXCore::Core::CpuStateFrame *Frame);
using JITCallback = void(*)(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP);
using AsmDispatch = void (*)(FEXCore::Core::CpuStateFrame* Frame);
using JITCallback = void (*)(FEXCore::Core::CpuStateFrame* Frame, uint64_t RIP);
AsmDispatch DispatchPtr;
JITCallback CallbackPtr;
private:
// Long division helpers
uint64_t LUDIVHandlerAddress{};
uint64_t LDIVHandlerAddress{};
uint64_t LUREMHandlerAddress{};
uint64_t LREMHandlerAddress{};
uint64_t LUDIVHandlerAddress {};
uint64_t LDIVHandlerAddress {};
uint64_t LUREMHandlerAddress {};
uint64_t LREMHandlerAddress {};
void EmitDispatcher();
};
}
} // namespace FEXCore::CPU
File diff suppressed because it is too large. Load diff
+31 -24
View File
@@ -21,10 +21,10 @@ class Decoder final {
public:
// New Frontend decoding
struct DecodedBlocks final {
uint64_t Entry{};
uint64_t NumInstructions{};
FEXCore::X86Tables::DecodedInst *DecodedInstructions;
bool HasInvalidInstruction{};
uint64_t Entry {};
uint64_t NumInstructions {};
FEXCore::X86Tables::DecodedInst* DecodedInstructions;
bool HasInvalidInstruction {};
};
struct DecodedBlockInformation final {
@@ -32,19 +32,24 @@ public:
fextl::vector<DecodedBlocks> Blocks;
};
Decoder(FEXCore::Context::ContextImpl *ctx);
Decoder(FEXCore::Context::ContextImpl* ctx);
~Decoder();
void DecodeInstructionsAtEntry(uint8_t const* InstStream, uint64_t PC, std::function<void(uint64_t BlockEntry, uint64_t Start, uint64_t Length)> AddContainedCodePage);
void DecodeInstructionsAtEntry(const uint8_t* InstStream, uint64_t PC, uint64_t MaxInst,
std::function<void(uint64_t BlockEntry, uint64_t Start, uint64_t Length)> AddContainedCodePage);
DecodedBlockInformation const *GetDecodedBlockInfo() const {
const DecodedBlockInformation* GetDecodedBlockInfo() const {
return &BlockInfo;
}
uint64_t DecodedMinAddress {};
uint64_t DecodedMaxAddress {~0ULL};
void SetSectionMaxAddress(uint64_t v) { SectionMaxAddress = v; }
void SetExternalBranches(fextl::set<uint64_t> *v) { ExternalBranches = v; }
void SetSectionMaxAddress(uint64_t v) {
SectionMaxAddress = v;
}
void SetExternalBranches(fextl::set<uint64_t>* v) {
ExternalBranches = v;
}
void DelayedDisownBuffer() {
PoolObject.DelayedDisownBuffer();
@@ -59,8 +64,8 @@ private:
bool L; // VEX.L bit (if set then 256 bit operation, if unset then scalar or 128-bit operation)
};
FEXCore::Context::ContextImpl *CTX;
const FEXCore::HLE::SyscallOSABI OSABI{};
FEXCore::Context::ContextImpl* CTX;
const FEXCore::HLE::SyscallOSABI OSABI {};
bool DecodeInstruction(uint64_t PC);
@@ -70,22 +75,24 @@ private:
uint8_t ReadByte();
uint8_t PeekByte(uint8_t Offset) const;
uint64_t ReadData(uint8_t Size);
void SkipBytes(uint8_t Size) { InstructionSize += Size; }
void SkipBytes(uint8_t Size) {
InstructionSize += Size;
}
bool NormalOp(FEXCore::X86Tables::X86InstInfo const *Info, uint16_t Op, DecodedHeader Options = {});
bool NormalOpHeader(FEXCore::X86Tables::X86InstInfo const *Info, uint16_t Op);
bool NormalOp(const FEXCore::X86Tables::X86InstInfo* Info, uint16_t Op, DecodedHeader Options = {});
bool NormalOpHeader(const FEXCore::X86Tables::X86InstInfo* Info, uint16_t Op);
static constexpr size_t DefaultDecodedBufferSize = 0x10000;
FEXCore::X86Tables::DecodedInst *DecodedBuffer{};
FEXCore::X86Tables::DecodedInst* DecodedBuffer {};
Utils::FixedSizePooledAllocation<FEXCore::X86Tables::DecodedInst*, 5000, 500> PoolObject;
size_t DecodedSize {};
uint8_t const *InstStream;
const uint8_t* InstStream;
static constexpr size_t MAX_INST_SIZE = 15;
uint8_t InstructionSize;
std::array<uint8_t, MAX_INST_SIZE> Instruction;
FEXCore::X86Tables::DecodedInst *DecodeInst;
FEXCore::X86Tables::DecodedInst* DecodeInst;
// This is for multiblock data tracking
bool SymbolAvailable {false};
@@ -99,21 +106,21 @@ private:
DecodedBlockInformation BlockInfo;
fextl::set<uint64_t> BlocksToDecode;
fextl::set<uint64_t> HasBlocks;
fextl::set<uint64_t> *ExternalBranches {nullptr};
fextl::set<uint64_t>* ExternalBranches {nullptr};
// ModRM rm decoding
using DecodeModRMPtr = void (FEXCore::Frontend::Decoder::*)(X86Tables::DecodedOperand *Operand, X86Tables::ModRMDecoded ModRM);
void DecodeModRM_16(X86Tables::DecodedOperand *Operand, X86Tables::ModRMDecoded ModRM);
void DecodeModRM_64(X86Tables::DecodedOperand *Operand, X86Tables::ModRMDecoded ModRM);
using DecodeModRMPtr = void (FEXCore::Frontend::Decoder::*)(X86Tables::DecodedOperand* Operand, X86Tables::ModRMDecoded ModRM);
void DecodeModRM_16(X86Tables::DecodedOperand* Operand, X86Tables::ModRMDecoded ModRM);
void DecodeModRM_64(X86Tables::DecodedOperand* Operand, X86Tables::ModRMDecoded ModRM);
static constexpr std::array<DecodeModRMPtr, 2> DecodeModRMs_Disp{
static constexpr std::array<DecodeModRMPtr, 2> DecodeModRMs_Disp {
&FEXCore::Frontend::Decoder::DecodeModRM_64,
&FEXCore::Frontend::Decoder::DecodeModRM_16,
};
const uint8_t *AdjustAddrForSpecialRegion(uint8_t const* _InstStream, uint64_t EntryPoint, uint64_t RIP);
const uint8_t* AdjustAddrForSpecialRegion(const uint8_t* _InstStream, uint64_t EntryPoint, uint64_t RIP);
FEXCORE_TELEMETRY_INIT(VEXOpTelem, TYPE_USES_VEX_OPS);
FEXCORE_TELEMETRY_INIT(EVEXOpTelem, TYPE_USES_EVEX_OPS);
};
}
} // namespace FEXCore::Frontend
-103
View File
@@ -1,103 +0,0 @@
// SPDX-License-Identifier: MIT
/*
$info$
tags: glue|gdbserver
$end_info$
*/
#pragma once
#include <FEXCore/Config/Config.h>
#include <FEXCore/Utils/Event.h>
#include <FEXCore/Utils/Threads.h>
#include <FEXCore/fextl/memory.h>
#include <FEXCore/fextl/string.h>
#include <atomic>
#include <istream>
#include <memory>
#include <mutex>
#include <stdint.h>
namespace FEXCore {
namespace Context {
class ContextImpl;
}
class GdbServer {
public:
GdbServer(FEXCore::Context::ContextImpl *ctx);
// Public for threading
void GdbServerLoop();
void AlertLibrariesChanged() {
LibraryMapChanged = true;
}
private:
void Break(int signal);
void OpenListenSocket();
fextl::unique_ptr<std::iostream> OpenSocket();
void StartThread();
fextl::string ReadPacket(std::iostream &stream);
void SendPacket(std::ostream &stream, const fextl::string& packet);
void SendACK(std::ostream &stream, bool NACK);
Event ThreadBreakEvent{};
void WaitForThreadWakeup();
struct HandledPacketType {
fextl::string Response{};
enum ResponseType {
TYPE_NONE,
TYPE_UNKNOWN,
TYPE_ACK,
TYPE_NACK,
TYPE_ONLYACK,
TYPE_ONLYNACK,
};
ResponseType TypeResponse{};
};
void SendPacketPair(const HandledPacketType& packetPair);
HandledPacketType ProcessPacket(const fextl::string &packet);
HandledPacketType handleQuery(const fextl::string &packet);
HandledPacketType handleXfer(const fextl::string &packet);
HandledPacketType handleMemory(const fextl::string &packet);
HandledPacketType handleV(const fextl::string& packet);
HandledPacketType handleThreadOp(const fextl::string &packet);
HandledPacketType handleBreakpoint(const fextl::string &packet);
HandledPacketType handleProgramOffsets();
HandledPacketType ThreadAction(char action, uint32_t tid);
fextl::string readRegs();
HandledPacketType readReg(const fextl::string& packet);
FEXCore::Context::ContextImpl *CTX;
fextl::unique_ptr<FEXCore::Threads::Thread> gdbServerThread;
fextl::unique_ptr<std::iostream> CommsStream;
std::mutex sendMutex;
bool SettingNoAckMode{false};
bool NoAckMode{false};
bool NonStopMode{false};
fextl::string ThreadString{};
fextl::string OSDataString{};
void buildLibraryMap();
std::atomic<bool> LibraryMapChanged = true;
fextl::string LibraryMapString{};
// Used to keep track of which signals to pass to the guest
std::array<bool, SignalDelegator::MAX_SIGNALS + 1> PassSignals{};
uint32_t CurrentDebuggingThread{};
int ListenSocket{};
FEX_CONFIG_OPT(Filename, APP_FILENAME);
};
}
+122 -174
View File
@@ -9,14 +9,6 @@
#ifdef _M_X86_64
#define XBYAK64
#define XBYAK_CUSTOM_ALLOC
#define XBYAK_CUSTOM_MALLOC FEXCore::Allocator::malloc
#define XBYAK_CUSTOM_FREE FEXCore::Allocator::free
#define XBYAK_CUSTOM_SETS
#define XBYAK_STD_UNORDERED_SET fextl::unordered_set
#define XBYAK_STD_UNORDERED_MAP fextl::unordered_map
#define XBYAK_STD_UNORDERED_MULTIMAP fextl::unordered_multimap
#define XBYAK_STD_LIST fextl::list
#define XBYAK_NO_EXCEPTION
#include <FEXCore/fextl/list.h>
#include <FEXCore/fextl/unordered_map.h>
@@ -36,24 +28,29 @@ namespace FEXCore {
[[maybe_unused]] constexpr uint32_t DCZID_BS_MASK = 0b0'1111;
#ifdef _M_ARM_64
[[maybe_unused]] static uint32_t GetDCZID() {
uint64_t Result{};
__asm("mrs %[Res], DCZID_EL0"
: [Res] "=r" (Result));
[[maybe_unused]]
static uint32_t GetDCZID() {
uint64_t Result {};
__asm("mrs %[Res], DCZID_EL0" : [Res] "=r"(Result));
return Result;
}
static uint32_t GetFPCR() {
uint64_t Result{};
__asm ("mrs %[Res], FPCR"
: [Res] "=r" (Result));
uint64_t Result {};
__asm("mrs %[Res], FPCR" : [Res] "=r"(Result));
return Result;
}
static void SetFPCR(uint64_t Value) {
__asm ("msr FPCR, %[Value]"
:: [Value] "r" (Value));
__asm("msr FPCR, %[Value]" ::[Value] "r"(Value));
}
static uint32_t GetMIDR() {
uint64_t Result {};
__asm("mrs %[Res], MIDR_EL1" : [Res] "=r"(Result));
return Result;
}
#else
static uint32_t GetDCZID() {
// Return unsupported
@@ -61,7 +58,7 @@ static uint32_t GetDCZID() {
}
#endif
static void OverrideFeatures(HostFeatures *Features) {
static void OverrideFeatures(HostFeatures* Features, uint64_t ForceSVEWidth) {
// Override features if the user has specifically called for it.
FEX_CONFIG_OPT(HostFeatures, HOSTFEATURES);
if (!HostFeatures()) {
@@ -69,151 +66,66 @@ static void OverrideFeatures(HostFeatures *Features) {
return;
}
const bool DisableAVX = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEAVX;
const bool EnableAVX = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEAVX;
LogMan::Throw::AFmt(!(DisableAVX && EnableAVX), "Disabling and Enabling CPU features are mutually exclusive");
#define ENABLE_DISABLE_OPTION(FeatureName, name, enum_name) \
do { \
const bool Disable##name = (HostFeatures() & FEXCore::Config::HostFeatures::DISABLE##enum_name) != 0; \
const bool Enable##name = (HostFeatures() & FEXCore::Config::HostFeatures::ENABLE##enum_name) != 0; \
LogMan::Throw::AFmt(!(Disable##name && Enable##name), "Disabling and Enabling CPU feature (" #name ") is mutually exclusive"); \
const bool AlreadyEnabled = Features->FeatureName; \
const bool Result = (AlreadyEnabled | Enable##name) & !Disable##name; \
Features->FeatureName = Result; \
} while (0)
const bool DisableAVX2 = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEAVX2;
const bool EnableAVX2 = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEAVX2;
LogMan::Throw::AFmt(!(DisableAVX2 && EnableAVX2), "Disabling and Enabling CPU features are mutually exclusive");
#define GET_SINGLE_OPTION(name, enum_name) \
const bool Disable##name = (HostFeatures() & FEXCore::Config::HostFeatures::DISABLE##enum_name) != 0; \
const bool Enable##name = (HostFeatures() & FEXCore::Config::HostFeatures::ENABLE##enum_name) != 0; \
LogMan::Throw::AFmt(!(Disable##name && Enable##name), "Disabling and Enabling CPU feature (" #name ") is mutually exclusive");
const bool DisableSVE = HostFeatures() & FEXCore::Config::HostFeatures::DISABLESVE;
const bool EnableSVE = HostFeatures() & FEXCore::Config::HostFeatures::ENABLESVE;
LogMan::Throw::AFmt(!(DisableSVE && EnableSVE), "Disabling and Enabling CPU features are mutually exclusive");
ENABLE_DISABLE_OPTION(SupportsAVX, AVX, AVX);
ENABLE_DISABLE_OPTION(SupportsSVE128, SVE, SVE);
ENABLE_DISABLE_OPTION(SupportsAFP, AFP, AFP);
ENABLE_DISABLE_OPTION(SupportsRCPC, LRCPC, LRCPC);
ENABLE_DISABLE_OPTION(SupportsTSOImm9, LRCPC2, LRCPC2);
ENABLE_DISABLE_OPTION(SupportsCSSC, CSSC, CSSC);
ENABLE_DISABLE_OPTION(SupportsPMULL_128Bit, PMULL128, PMULL128);
ENABLE_DISABLE_OPTION(SupportsRAND, RNG, RNG);
ENABLE_DISABLE_OPTION(SupportsCLZERO, CLZERO, CLZERO);
ENABLE_DISABLE_OPTION(SupportsAtomics, Atomics, ATOMICS);
ENABLE_DISABLE_OPTION(SupportsFCMA, FCMA, FCMA);
ENABLE_DISABLE_OPTION(SupportsFlagM, FlagM, FLAGM);
ENABLE_DISABLE_OPTION(SupportsFlagM2, FlagM2, FLAGM2);
ENABLE_DISABLE_OPTION(SupportsRPRES, RPRES, RPRES);
ENABLE_DISABLE_OPTION(SupportsPreserveAllABI, PRESERVEALLABI, PRESERVEALLABI);
GET_SINGLE_OPTION(Crypto, CRYPTO);
const bool DisableAFP = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEAFP;
const bool EnableAFP = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEAFP;
LogMan::Throw::AFmt(!(DisableAFP && EnableAFP), "Disabling and Enabling CPU features are mutually exclusive");
#undef ENABLE_DISABLE_OPTION
#undef GET_SINGLE_OPTION
const bool DisableLRCPC = HostFeatures() & FEXCore::Config::HostFeatures::DISABLELRCPC;
const bool EnableLRCPC = HostFeatures() & FEXCore::Config::HostFeatures::ENABLELRCPC;
LogMan::Throw::AFmt(!(DisableLRCPC && EnableLRCPC), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableLRCPC2 = HostFeatures() & FEXCore::Config::HostFeatures::DISABLELRCPC2;
const bool EnableLRCPC2 = HostFeatures() & FEXCore::Config::HostFeatures::ENABLELRCPC2;
LogMan::Throw::AFmt(!(DisableLRCPC2 && EnableLRCPC2), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableCSSC = HostFeatures() & FEXCore::Config::HostFeatures::DISABLECSSC;
const bool EnableCSSC = HostFeatures() & FEXCore::Config::HostFeatures::ENABLECSSC;
LogMan::Throw::AFmt(!(DisableCSSC && EnableCSSC), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisablePMULL128 = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEPMULL128;
const bool EnablePMULL128 = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEPMULL128;
LogMan::Throw::AFmt(!(DisablePMULL128 && EnablePMULL128), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableRNG = HostFeatures() & FEXCore::Config::HostFeatures::DISABLERNG;
const bool EnableRNG = HostFeatures() & FEXCore::Config::HostFeatures::ENABLERNG;
LogMan::Throw::AFmt(!(DisableRNG && EnableRNG), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableCLZERO = HostFeatures() & FEXCore::Config::HostFeatures::DISABLECLZERO;
const bool EnableCLZERO = HostFeatures() & FEXCore::Config::HostFeatures::ENABLECLZERO;
LogMan::Throw::AFmt(!(DisableCLZERO && EnableCLZERO), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableAtomics = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEATOMICS;
const bool EnableAtomics = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEATOMICS;
LogMan::Throw::AFmt(!(DisableAtomics && EnableAtomics), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableFCMA = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEFCMA;
const bool EnableFCMA = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEFCMA;
LogMan::Throw::AFmt(!(DisableFCMA && EnableFCMA), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableFlagM = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEFLAGM;
const bool EnableFlagM = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEFLAGM;
LogMan::Throw::AFmt(!(DisableFlagM && EnableFlagM), "Disabling and Enabling CPU features are mutually exclusive");
const bool DisableFlagM2 = HostFeatures() & FEXCore::Config::HostFeatures::DISABLEFLAGM2;
const bool EnableFlagM2 = HostFeatures() & FEXCore::Config::HostFeatures::ENABLEFLAGM2;
LogMan::Throw::AFmt(!(DisableFlagM2 && EnableFlagM2), "Disabling and Enabling CPU features are mutually exclusive");
if (EnableAVX) {
Features->SupportsAVX = true;
}
else if (DisableAVX) {
Features->SupportsAVX = false;
}
if (EnableAVX2) {
Features->SupportsAVX2 = true;
}
else if (DisableAVX2) {
Features->SupportsAVX2 = false;
}
if (EnableSVE) {
Features->SupportsSVE = true;
}
else if (DisableSVE) {
Features->SupportsSVE = false;
}
if (EnableAFP) {
Features->SupportsFlushInputsToZero = true;
}
else if (DisableAFP) {
Features->SupportsFlushInputsToZero = false;
}
if (EnableLRCPC) {
Features->SupportsRCPC = true;
}
else if (DisableLRCPC) {
Features->SupportsRCPC = false;
}
if (EnableLRCPC2) {
Features->SupportsTSOImm9 = true;
}
else if (DisableLRCPC2) {
Features->SupportsTSOImm9 = false;
}
if (EnableCSSC) {
Features->SupportsCSSC = true;
}
else if (DisableCSSC) {
Features->SupportsCSSC = false;
}
if (EnablePMULL128) {
if (EnableCrypto) {
Features->SupportsAES = true;
Features->SupportsCRC = true;
Features->SupportsSHA = true;
Features->SupportsPMULL_128Bit = true;
}
else if (DisablePMULL128) {
Features->SupportsAES256 = true;
} else if (DisableCrypto) {
Features->SupportsAES = false;
Features->SupportsCRC = false;
Features->SupportsSHA = false;
Features->SupportsPMULL_128Bit = false;
Features->SupportsAES256 = false;
}
if (EnableRNG) {
Features->SupportsRAND = true;
}
else if (DisableRNG) {
Features->SupportsRAND = false;
}
if (EnableCLZERO) {
Features->SupportsCLZERO = true;
}
else if (DisableCLZERO) {
Features->SupportsCLZERO = false;
}
if (EnableAtomics) {
Features->SupportsAtomics = true;
}
else if (DisableAtomics) {
Features->SupportsAtomics = false;
}
if (EnableFCMA) {
Features->SupportsFCMA = true;
}
else if (DisableFCMA) {
Features->SupportsFCMA = false;
}
if (EnableFlagM) {
Features->SupportsFlagM = true;
}
else if (DisableFlagM) {
Features->SupportsFlagM = false;
}
if (EnableFlagM2) {
Features->SupportsFlagM2 = true;
}
else if (DisableFlagM2) {
Features->SupportsFlagM2 = false;
}
///< Only force enable SVE256 if SVE is already enabled and ForceSVEWidth is set to >= 256.
Features->SupportsSVE256 = ForceSVEWidth && ForceSVEWidth >= 256;
}
HostFeatures::HostFeatures() {
#ifdef VIXL_SIMULATOR
auto Features = vixl::CPUFeatures::All();
// Vixl simulator doesn't support AFP.
Features.Remove(vixl::CPUFeatures::Feature::kAFP);
// Vixl simulator doesn't support RPRES.
Features.Remove(vixl::CPUFeatures::Feature::kRPRES);
#elif !defined(_WIN32)
auto Features = vixl::CPUFeatures::InferFromOS();
#else
@@ -221,13 +133,17 @@ HostFeatures::HostFeatures() {
auto Features = vixl::CPUFeatures::InferFromIDRegisters();
#endif
FEX_CONFIG_OPT(ForceSVEWidth, FORCESVEWIDTH);
FEX_CONFIG_OPT(Is64BitMode, IS64BIT_MODE);
SupportsAES = Features.Has(vixl::CPUFeatures::Feature::kAES);
SupportsCRC = Features.Has(vixl::CPUFeatures::Feature::kCRC32);
SupportsSHA = Features.Has(vixl::CPUFeatures::Feature::kSHA1) && Features.Has(vixl::CPUFeatures::Feature::kSHA2);
SupportsAtomics = Features.Has(vixl::CPUFeatures::Feature::kAtomics);
SupportsRAND = Features.Has(vixl::CPUFeatures::Feature::kRNG);
// Only supported when FEAT_AFP is supported
SupportsFlushInputsToZero = Features.Has(vixl::CPUFeatures::Feature::kAFP);
SupportsAFP = Features.Has(vixl::CPUFeatures::Feature::kAFP);
SupportsRCPC = Features.Has(vixl::CPUFeatures::Feature::kRCpc);
SupportsTSOImm9 = Features.Has(vixl::CPUFeatures::Feature::kRCpcImm);
SupportsPMULL_128Bit = Features.Has(vixl::CPUFeatures::Feature::kPmull1Q);
@@ -235,21 +151,23 @@ HostFeatures::HostFeatures() {
SupportsFCMA = Features.Has(vixl::CPUFeatures::Feature::kFcma);
SupportsFlagM = Features.Has(vixl::CPUFeatures::Feature::kFlagM);
SupportsFlagM2 = Features.Has(vixl::CPUFeatures::Feature::kAXFlag);
SupportsRPRES = Features.Has(vixl::CPUFeatures::Feature::kRPRES);
Supports3DNow = true;
SupportsSSE4A = true;
#ifdef VIXL_SIMULATOR
// Hardcode enable SVE with 256-bit wide registers.
SupportsSVE = true;
SupportsAVX = true;
SupportsSVE128 = ForceSVEWidth() ? ForceSVEWidth() >= 128 : true;
SupportsSVE256 = ForceSVEWidth() ? ForceSVEWidth() >= 256 : true;
#else
SupportsSVE = Features.Has(vixl::CPUFeatures::Feature::kSVE);
SupportsAVX = Features.Has(vixl::CPUFeatures::Feature::kSVE2) &&
vixl::aarch64::CPU::ReadSVEVectorLengthInBits() >= 256;
SupportsSVE128 = Features.Has(vixl::CPUFeatures::Feature::kSVE2);
SupportsSVE256 = Features.Has(vixl::CPUFeatures::Feature::kSVE2) && vixl::aarch64::CPU::ReadSVEVectorLengthInBits() >= 256;
#endif
// TODO: AVX2 is currently unsupported. Disable until the remaining features are implemented.
SupportsAVX2 = false;
SupportsSHA = true;
SupportsAVX = true;
SupportsAES256 = SupportsAVX && SupportsAES;
SupportsBMI1 = true;
SupportsBMI2 = true;
SupportsCLWB = true;
@@ -262,21 +180,19 @@ HostFeatures::HostFeatures() {
// We need to get the CPU's cache line size
// We expect sane targets that have correct cacheline sizes across clusters
uint64_t CTR;
__asm volatile ("mrs %[ctr], ctr_el0"
: [ctr] "=r"(CTR));
__asm volatile("mrs %[ctr], ctr_el0" : [ctr] "=r"(CTR));
DCacheLineSize = 4 << ((CTR >> 16) & 0xF);
ICacheLineSize = 4 << (CTR & 0xF);
// Test if this CPU supports float exception trapping by attempting to enable
// On unsupported these bits are architecturally defined as RAZ/WI
constexpr uint32_t ExceptionEnableTraps =
(1U << 8) | // Invalid Operation float exception trap enable
(1U << 9) | // Divide by zero float exception trap enable
(1U << 10) | // Overflow float exception trap enable
(1U << 11) | // Underflow float exception trap enable
(1U << 12) | // Inexact float exception trap enable
(1U << 15); // Input Denormal float exception trap enable
constexpr uint32_t ExceptionEnableTraps = (1U << 8) | // Invalid Operation float exception trap enable
(1U << 9) | // Divide by zero float exception trap enable
(1U << 10) | // Overflow float exception trap enable
(1U << 11) | // Underflow float exception trap enable
(1U << 12) | // Inexact float exception trap enable
(1U << 15); // Input Denormal float exception trap enable
uint32_t OriginalFPCR = GetFPCR();
uint32_t FPCR = OriginalFPCR | ExceptionEnableTraps;
@@ -286,11 +202,31 @@ HostFeatures::HostFeatures() {
// Set FPCR back to original just in case anything changed
SetFPCR(OriginalFPCR);
if (SupportsRAND) {
const auto MIDR = GetMIDR();
constexpr uint32_t Implementer_QCOM = 0x51;
constexpr uint32_t PartNum_Oryon1 = 0x001;
const uint32_t MIDR_Implementer = (MIDR >> 24) & 0xFF;
const uint32_t MIDR_PartNum = (MIDR >> 4) & 0xFFF;
if (MIDR_Implementer == Implementer_QCOM && MIDR_PartNum == PartNum_Oryon1) {
// Work around an errata in Qualcomm's Oryon.
// While this CPU implements the RAND extension:
// - The RNDR register works.
// - The RNDRRS register will never read a random number. (Always return failure)
// This is contrary to x86 RNG behaviour where it allows spurious failure with RDSEED, but guarantees eventual success.
// This manifested itself on Linux when an x86 processor failed to guarantee forward progress and boot of services would infinite
// loop. Just disable this extension if this CPU is detected.
SupportsRAND = false;
}
}
#endif
#ifdef VIXL_SIMULATOR
// simulator doesn't support dc(ZVA)
SupportsCLZERO = false;
// Simulator doesn't support SHA
SupportsSHA = false;
#else
// Check if we can support cacheline clears
uint32_t DCZID = GetDCZID();
@@ -309,7 +245,7 @@ HostFeatures::HostFeatures() {
ICacheLineSize = 64U;
#if !defined(VIXL_SIMULATOR)
Xbyak::util::Cpu X86Features{};
Xbyak::util::Cpu X86Features {};
SupportsAES = X86Features.has(Xbyak::util::Cpu::tAESNI);
SupportsCRC = X86Features.has(Xbyak::util::Cpu::tSSE42);
SupportsRAND = X86Features.has(Xbyak::util::Cpu::tRDRAND) && X86Features.has(Xbyak::util::Cpu::tRDSEED);
@@ -318,12 +254,12 @@ HostFeatures::HostFeatures() {
Supports3DNow = X86Features.has(Xbyak::util::Cpu::t3DN) && X86Features.has(Xbyak::util::Cpu::tE3DN);
SupportsSSE4A = X86Features.has(Xbyak::util::Cpu::tSSE4a);
SupportsAVX = true;
SupportsAVX2 = true;
SupportsSHA = X86Features.has(Xbyak::util::Cpu::tSHA);
SupportsBMI1 = X86Features.has(Xbyak::util::Cpu::tBMI1);
SupportsBMI2 = X86Features.has(Xbyak::util::Cpu::tBMI2);
SupportsCLWB = X86Features.has(Xbyak::util::Cpu::tCLWB);
SupportsPMULL_128Bit = X86Features.has(Xbyak::util::Cpu::tPCLMULQDQ);
SupportsAES256 = SupportsAES && X86Features.has(Xbyak::util::Cpu::tVAES);
// xbyak doesn't know how to check for CLZero
// First ensure we support a new enough extended CPUID function range
@@ -336,10 +272,22 @@ HostFeatures::HostFeatures() {
SupportsCLZERO = data[1] & 1;
}
SupportsFlushInputsToZero = true;
SupportsAFP = true;
SupportsFloatExceptions = true;
#endif
#endif
OverrideFeatures(this);
}
SupportsPreserveAllABI = FEXCORE_HAS_PRESERVE_ALL_ATTR;
if (!Is64BitMode()) {
///< Always disable AVX and AVX2 in 32-bit mode.
// When AVX256 is enabled, signal frames start using significantly more stack space.
// - 16bytes * 16 registers = 256 bytes for XMM registers.
// - 32bytes * 16 registers = 512 bytes for YMM registers.
// There are known game failures on real x86 hardware where a 32-bit game is running up against the wall on stack space on non-AVX
// hardware and then explodes when run on AVX hardware. This is to guard against that.
SupportsAVX = false;
}
OverrideFeatures(this, ForceSVEWidth());
}
} // namespace FEXCore
@@ -1,2 +0,0 @@
// SPDX-License-Identifier: MIT
#include <FEXCore/Debug/InternalThreadState.h>
@@ -2,48 +2,36 @@
#include "Common/SoftFloat.h"
#include "Common/SoftFloat-3e/softfloat.h"
#include <FEXCore/IR/IR.h>
#include "Interface/Core/Interpreter/Fallbacks/FallbackOpHandler.h"
#include "Interface/IR/IR.h"
namespace FEXCore::CPU {
FEXCORE_PRESERVE_ALL_ATTR
static void LoadDeferredFCW(uint16_t NewFCW) {
FEXCORE_PRESERVE_ALL_ATTR static void LoadDeferredFCW(uint16_t NewFCW) {
auto PC = (NewFCW >> 8) & 3;
switch(PC) {
case 0: extF80_roundingPrecision = 32; break;
case 2: extF80_roundingPrecision = 64; break;
case 3: extF80_roundingPrecision = 80; break;
case 1: LOGMAN_MSG_A_FMT("Invalid x87 precision mode, {}", PC);
switch (PC) {
case 0: extF80_roundingPrecision = 32; break;
case 2: extF80_roundingPrecision = 64; break;
case 3: extF80_roundingPrecision = 80; break;
case 1: LOGMAN_MSG_A_FMT("Invalid x87 precision mode, {}", PC);
}
auto RC = (NewFCW >> 10) & 3;
switch(RC) {
case 0:
softfloat_roundingMode = softfloat_round_near_even;
break;
case 1:
softfloat_roundingMode = softfloat_round_min;
break;
case 2:
softfloat_roundingMode = softfloat_round_max;
break;
case 3:
softfloat_roundingMode = softfloat_round_minMag;
break;
switch (RC) {
case 0: softfloat_roundingMode = softfloat_round_near_even; break;
case 1: softfloat_roundingMode = softfloat_round_min; break;
case 2: softfloat_roundingMode = softfloat_round_max; break;
case 3: softfloat_roundingMode = softfloat_round_minMag; break;
}
}
template<>
struct OpHandlers<IR::OP_F80CVTTO> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle4(uint16_t NewFCW, float src) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle4(uint16_t NewFCW, float src) {
LoadDeferredFCW(NewFCW);
return src;
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle8(uint16_t NewFCW, double src) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle8(uint16_t NewFCW, double src) {
LoadDeferredFCW(NewFCW);
return src;
}
@@ -52,24 +40,20 @@ struct OpHandlers<IR::OP_F80CVTTO> {
template<>
struct OpHandlers<IR::OP_F80CMP> {
template<uint32_t Flags>
FEXCORE_PRESERVE_ALL_ATTR
static uint64_t handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static uint64_t handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
bool eq, lt, nan;
uint64_t ResultFlags = 0;
X80SoftFloat::FCMP(Src1, Src2, &eq, &lt, &nan);
if (Flags & (1 << IR::FCMP_FLAG_LT) &&
lt) {
if (Flags & (1 << IR::FCMP_FLAG_LT) && lt) {
ResultFlags |= (1 << IR::FCMP_FLAG_LT);
}
if (Flags & (1 << IR::FCMP_FLAG_UNORDERED) &&
nan) {
if (Flags & (1 << IR::FCMP_FLAG_UNORDERED) && nan) {
ResultFlags |= (1 << IR::FCMP_FLAG_UNORDERED);
}
if (Flags & (1 << IR::FCMP_FLAG_EQ) &&
eq) {
if (Flags & (1 << IR::FCMP_FLAG_EQ) && eq) {
ResultFlags |= (1 << IR::FCMP_FLAG_EQ);
}
return ResultFlags;
@@ -78,14 +62,12 @@ struct OpHandlers<IR::OP_F80CMP> {
template<>
struct OpHandlers<IR::OP_F80CVT> {
FEXCORE_PRESERVE_ALL_ATTR
static float handle4(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static float handle4(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
return src;
}
FEXCORE_PRESERVE_ALL_ATTR
static double handle8(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static double handle8(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
return src;
}
@@ -93,26 +75,22 @@ struct OpHandlers<IR::OP_F80CVT> {
template<>
struct OpHandlers<IR::OP_F80CVTINT> {
FEXCORE_PRESERVE_ALL_ATTR
static int16_t handle2(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static int16_t handle2(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
return src;
}
FEXCORE_PRESERVE_ALL_ATTR
static int32_t handle4(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static int32_t handle4(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
return src;
}
FEXCORE_PRESERVE_ALL_ATTR
static int64_t handle8(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static int64_t handle8(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
return src;
}
FEXCORE_PRESERVE_ALL_ATTR
static int16_t handle2t(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static int16_t handle2t(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
auto rv = extF80_to_i32(src, softfloat_round_minMag, false);
@@ -125,14 +103,12 @@ struct OpHandlers<IR::OP_F80CVTINT> {
}
}
FEXCORE_PRESERVE_ALL_ATTR
static int32_t handle4t(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static int32_t handle4t(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
return extF80_to_i32(src, softfloat_round_minMag, false);
}
FEXCORE_PRESERVE_ALL_ATTR
static int64_t handle8t(uint16_t NewFCW, X80SoftFloat src) {
FEXCORE_PRESERVE_ALL_ATTR static int64_t handle8t(uint16_t NewFCW, X80SoftFloat src) {
LoadDeferredFCW(NewFCW);
return extF80_to_i64(src, softfloat_round_minMag, false);
}
@@ -140,14 +116,12 @@ struct OpHandlers<IR::OP_F80CVTINT> {
template<>
struct OpHandlers<IR::OP_F80CVTTOINT> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle2(uint16_t NewFCW, int16_t src) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle2(uint16_t NewFCW, int16_t src) {
LoadDeferredFCW(NewFCW);
return src;
}
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle4(uint16_t NewFCW, int32_t src) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle4(uint16_t NewFCW, int32_t src) {
LoadDeferredFCW(NewFCW);
return src;
}
@@ -155,8 +129,7 @@ struct OpHandlers<IR::OP_F80CVTTOINT> {
template<>
struct OpHandlers<IR::OP_F80ROUND> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FRNDINT(Src1);
}
@@ -164,8 +137,7 @@ struct OpHandlers<IR::OP_F80ROUND> {
template<>
struct OpHandlers<IR::OP_F80F2XM1> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::F2XM1(Src1);
}
@@ -173,8 +145,7 @@ struct OpHandlers<IR::OP_F80F2XM1> {
template<>
struct OpHandlers<IR::OP_F80TAN> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FTAN(Src1);
}
@@ -182,8 +153,7 @@ struct OpHandlers<IR::OP_F80TAN> {
template<>
struct OpHandlers<IR::OP_F80SQRT> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FSQRT(Src1);
}
@@ -191,8 +161,7 @@ struct OpHandlers<IR::OP_F80SQRT> {
template<>
struct OpHandlers<IR::OP_F80SIN> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FSIN(Src1);
}
@@ -200,8 +169,7 @@ struct OpHandlers<IR::OP_F80SIN> {
template<>
struct OpHandlers<IR::OP_F80COS> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FCOS(Src1);
}
@@ -209,8 +177,7 @@ struct OpHandlers<IR::OP_F80COS> {
template<>
struct OpHandlers<IR::OP_F80XTRACT_EXP> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FXTRACT_EXP(Src1);
}
@@ -218,8 +185,7 @@ struct OpHandlers<IR::OP_F80XTRACT_EXP> {
template<>
struct OpHandlers<IR::OP_F80XTRACT_SIG> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FXTRACT_SIG(Src1);
}
@@ -227,8 +193,7 @@ struct OpHandlers<IR::OP_F80XTRACT_SIG> {
template<>
struct OpHandlers<IR::OP_F80ADD> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FADD(Src1, Src2);
}
@@ -236,8 +201,7 @@ struct OpHandlers<IR::OP_F80ADD> {
template<>
struct OpHandlers<IR::OP_F80SUB> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FSUB(Src1, Src2);
}
@@ -245,8 +209,7 @@ struct OpHandlers<IR::OP_F80SUB> {
template<>
struct OpHandlers<IR::OP_F80MUL> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FMUL(Src1, Src2);
}
@@ -254,8 +217,7 @@ struct OpHandlers<IR::OP_F80MUL> {
template<>
struct OpHandlers<IR::OP_F80DIV> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FDIV(Src1, Src2);
}
@@ -263,8 +225,7 @@ struct OpHandlers<IR::OP_F80DIV> {
template<>
struct OpHandlers<IR::OP_F80FYL2X> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FYL2X(Src1, Src2);
}
@@ -272,8 +233,7 @@ struct OpHandlers<IR::OP_F80FYL2X> {
template<>
struct OpHandlers<IR::OP_F80ATAN> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FATAN(Src1, Src2);
}
@@ -281,8 +241,7 @@ struct OpHandlers<IR::OP_F80ATAN> {
template<>
struct OpHandlers<IR::OP_F80FPREM1> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FREM1(Src1, Src2);
}
@@ -290,8 +249,7 @@ struct OpHandlers<IR::OP_F80FPREM1> {
template<>
struct OpHandlers<IR::OP_F80FPREM> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FREM(Src1, Src2);
}
@@ -299,8 +257,7 @@ struct OpHandlers<IR::OP_F80FPREM> {
template<>
struct OpHandlers<IR::OP_F80SCALE> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1, X80SoftFloat Src2) {
LoadDeferredFCW(NewFCW);
return X80SoftFloat::FSCALE(Src1, Src2);
}
@@ -374,15 +331,14 @@ template<>
struct OpHandlers<IR::OP_F64SCALE> {
static double handle(uint16_t NewFCW, double src1, double src2) {
LoadDeferredFCW(NewFCW);
double trunc = (double)(int64_t)(src2); //truncate
double trunc = (double)(int64_t)(src2); // truncate
return src1 * exp2(trunc);
}
};
template<>
struct OpHandlers<IR::OP_F80BCDSTORE> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src1) {
LoadDeferredFCW(NewFCW);
bool Negative = Src1.Sign;
@@ -393,7 +349,7 @@ struct OpHandlers<IR::OP_F80BCDSTORE> {
uint64_t Tmp = Src1;
X80SoftFloat Rv;
uint8_t *BCD = reinterpret_cast<uint8_t*>(&Rv);
uint8_t* BCD = reinterpret_cast<uint8_t*>(&Rv);
memset(BCD, 0, 10);
for (size_t i = 0; i < 9; ++i) {
@@ -423,11 +379,10 @@ struct OpHandlers<IR::OP_F80BCDSTORE> {
template<>
struct OpHandlers<IR::OP_F80BCDLOAD> {
FEXCORE_PRESERVE_ALL_ATTR
static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src) {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t NewFCW, X80SoftFloat Src) {
LoadDeferredFCW(NewFCW);
uint8_t *Src1 = reinterpret_cast<uint8_t *>(&Src);
uint64_t BCD{};
uint8_t* Src1 = reinterpret_cast<uint8_t*>(&Src);
uint64_t BCD {};
// We walk through each uint8_t and pull out the BCD encoding
// Each 4bit split is a digit
// Only 0-9 is supported, A-F results in undefined data
@@ -68,8 +68,7 @@ namespace FEXCore::CPU {
//
// 5. Done.
//
template <IR::IROps Op>
struct OpHandlers {
};
template<IR::IROps Op>
struct OpHandlers {};
} // namespace FEXCore::CPU
@@ -10,23 +10,23 @@
namespace FEXCore::CPU {
template<typename R, typename... Args>
static FallbackInfo GetFallbackInfo(R(*fn)(Args...), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
static FallbackInfo GetFallbackInfo(R (*fn)(Args...), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_UNKNOWN, (void*)fn, HandlerIndex, false};
}
template<>
FallbackInfo GetFallbackInfo(double(*fn)(uint16_t, double), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
FallbackInfo GetFallbackInfo(double (*fn)(uint16_t, double), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F64_I16_F64, (void*)fn, HandlerIndex, false};
}
template<>
FallbackInfo GetFallbackInfo(double(*fn)(uint16_t, double,double), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
FallbackInfo GetFallbackInfo(double (*fn)(uint16_t, double, double), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F64_I16_F64_F64, (void*)fn, HandlerIndex, false};
}
void InterpreterOps::FillFallbackIndexPointers(uint64_t *Info) {
Info[Core::OPINDEX_F80CVTTO_4] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4);
Info[Core::OPINDEX_F80CVTTO_8] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8);
void InterpreterOps::FillFallbackIndexPointers(uint64_t* Info) {
Info[Core::OPINDEX_F80CVTTO_4] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4);
Info[Core::OPINDEX_F80CVTTO_8] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8);
Info[Core::OPINDEX_F80CVT_4] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle4);
Info[Core::OPINDEX_F80CVT_8] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle8);
Info[Core::OPINDEX_F80CVTINT_2] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2);
@@ -55,8 +55,8 @@ void InterpreterOps::FillFallbackIndexPointers(uint64_t *Info) {
Info[Core::OPINDEX_F80COS] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80COS>::handle);
Info[Core::OPINDEX_F80XTRACT_EXP] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80XTRACT_EXP>::handle);
Info[Core::OPINDEX_F80XTRACT_SIG] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80XTRACT_SIG>::handle);
Info[Core::OPINDEX_F80BCDSTORE] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80BCDSTORE>::handle);
Info[Core::OPINDEX_F80BCDLOAD] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80BCDLOAD>::handle);
Info[Core::OPINDEX_F80BCDSTORE] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80BCDSTORE>::handle);
Info[Core::OPINDEX_F80BCDLOAD] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80BCDLOAD>::handle);
// Binary
Info[Core::OPINDEX_F80ADD] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_F80ADD>::handle);
@@ -85,126 +85,123 @@ void InterpreterOps::FillFallbackIndexPointers(uint64_t *Info) {
Info[Core::OPINDEX_VPCMPISTRX] = reinterpret_cast<uint64_t>(&FEXCore::CPU::OpHandlers<IR::OP_VPCMPISTRX>::handle);
}
bool InterpreterOps::GetFallbackHandler(IR::IROp_Header const *IROp, FallbackInfo *Info) {
bool InterpreterOps::GetFallbackHandler(bool SupportsPreserveAllABI, const IR::IROp_Header* IROp, FallbackInfo* Info) {
uint8_t OpSize = IROp->Size;
switch(IROp->Op) {
case IR::OP_F80CVTTO: {
auto Op = IROp->C<IR::IROp_F80CVTTo>();
switch (IROp->Op) {
case IR::OP_F80CVTTO: {
auto Op = IROp->C<IR::IROp_F80CVTTo>();
switch (Op->SrcSize) {
case 4: {
*Info = {FABI_F80_I16_F32, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4, Core::OPINDEX_F80CVTTO_4, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
}
case 8: {
*Info = {FABI_F80_I16_F64, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8, Core::OPINDEX_F80CVTTO_8, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVT: {
switch (OpSize) {
case 4: {
*Info = {FABI_F32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle4, Core::OPINDEX_F80CVT_4, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
}
case 8: {
*Info = {FABI_F64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle8, Core::OPINDEX_F80CVT_8, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVTINT: {
auto Op = IROp->C<IR::IROp_F80CVTInt>();
switch (OpSize) {
case 2: {
if (Op->Truncate) {
*Info = {FABI_I16_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2t, Core::OPINDEX_F80CVTINT_TRUNC2, FEXCORE_HAS_PRESERVE_ALL_ATTR};
}
else {
*Info = {FABI_I16_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2, Core::OPINDEX_F80CVTINT_2, FEXCORE_HAS_PRESERVE_ALL_ATTR};
}
return true;
}
case 4: {
if (Op->Truncate) {
*Info = {FABI_I32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4t, Core::OPINDEX_F80CVTINT_TRUNC4, FEXCORE_HAS_PRESERVE_ALL_ATTR};
}
else {
*Info = {FABI_I32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4, Core::OPINDEX_F80CVTINT_4, FEXCORE_HAS_PRESERVE_ALL_ATTR};
}
return true;
}
case 8: {
if (Op->Truncate) {
*Info = {FABI_I64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8t, Core::OPINDEX_F80CVTINT_TRUNC8, FEXCORE_HAS_PRESERVE_ALL_ATTR};
}
else {
*Info = {FABI_I64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8, Core::OPINDEX_F80CVTINT_8, FEXCORE_HAS_PRESERVE_ALL_ATTR};
}
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CMP: {
auto Op = IROp->C<IR::IROp_F80Cmp>();
static constexpr std::array handlers{
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<0>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<1>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<2>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<3>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<4>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<5>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<6>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<7>,
};
*Info = {FABI_I64_I16_F80_F80, (void*)handlers[Op->Flags], (Core::FallbackHandlerIndex)(Core::OPINDEX_F80CMP_0 + Op->Flags), FEXCORE_HAS_PRESERVE_ALL_ATTR};
switch (Op->SrcSize) {
case 4: {
*Info = {FABI_F80_I16_F32, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4, Core::OPINDEX_F80CVTTO_4, SupportsPreserveAllABI};
return true;
}
case 8: {
*Info = {FABI_F80_I16_F64, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8, Core::OPINDEX_F80CVTTO_8, SupportsPreserveAllABI};
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVT: {
switch (OpSize) {
case 4: {
*Info = {FABI_F32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle4, Core::OPINDEX_F80CVT_4, SupportsPreserveAllABI};
return true;
}
case 8: {
*Info = {FABI_F64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle8, Core::OPINDEX_F80CVT_8, SupportsPreserveAllABI};
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVTINT: {
auto Op = IROp->C<IR::IROp_F80CVTInt>();
case IR::OP_F80CVTTOINT: {
auto Op = IROp->C<IR::IROp_F80CVTToInt>();
switch (Op->SrcSize) {
case 2: {
*Info = {FABI_F80_I16_I16, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle2, Core::OPINDEX_F80CVTTOINT_2, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
}
case 4: {
*Info = {FABI_F80_I16_I32, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle4, Core::OPINDEX_F80CVTTOINT_4, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
switch (OpSize) {
case 2: {
if (Op->Truncate) {
*Info = {FABI_I16_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2t, Core::OPINDEX_F80CVTINT_TRUNC2,
SupportsPreserveAllABI};
} else {
*Info = {FABI_I16_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2, Core::OPINDEX_F80CVTINT_2, SupportsPreserveAllABI};
}
break;
return true;
}
case 4: {
if (Op->Truncate) {
*Info = {FABI_I32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4t, Core::OPINDEX_F80CVTINT_TRUNC4,
SupportsPreserveAllABI};
} else {
*Info = {FABI_I32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4, Core::OPINDEX_F80CVTINT_4, SupportsPreserveAllABI};
}
return true;
}
case 8: {
if (Op->Truncate) {
*Info = {FABI_I64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8t, Core::OPINDEX_F80CVTINT_TRUNC8,
SupportsPreserveAllABI};
} else {
*Info = {FABI_I64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8, Core::OPINDEX_F80CVTINT_8, SupportsPreserveAllABI};
}
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CMP: {
auto Op = IROp->C<IR::IROp_F80Cmp>();
#define COMMON_UNARY_X87_OP(OP) \
case IR::OP_F80##OP: { \
*Info = {FABI_F80_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80##OP>::handle, Core::OPINDEX_F80##OP, FEXCORE_HAS_PRESERVE_ALL_ATTR}; \
return true; \
}
static constexpr std::array handlers {
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<0>, &FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<1>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<2>, &FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<3>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<4>, &FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<5>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<6>, &FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<7>,
};
#define COMMON_BINARY_X87_OP(OP) \
case IR::OP_F80##OP: { \
*Info = {FABI_F80_I16_F80_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80##OP>::handle, Core::OPINDEX_F80##OP, FEXCORE_HAS_PRESERVE_ALL_ATTR}; \
return true; \
}
*Info = {FABI_I64_I16_F80_F80, (void*)handlers[Op->Flags], (Core::FallbackHandlerIndex)(Core::OPINDEX_F80CMP_0 + Op->Flags),
SupportsPreserveAllABI};
return true;
}
#define COMMON_F64_OP(OP) \
case IR::OP_F64##OP: { \
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F64##OP>::handle, Core::OPINDEX_F64##OP); \
return true; \
case IR::OP_F80CVTTOINT: {
auto Op = IROp->C<IR::IROp_F80CVTToInt>();
switch (Op->SrcSize) {
case 2: {
*Info = {FABI_F80_I16_I16, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle2, Core::OPINDEX_F80CVTTOINT_2, SupportsPreserveAllABI};
return true;
}
case 4: {
*Info = {FABI_F80_I16_I32, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle4, Core::OPINDEX_F80CVTTOINT_4, SupportsPreserveAllABI};
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
#define COMMON_UNARY_X87_OP(OP) \
case IR::OP_F80##OP: { \
*Info = {FABI_F80_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80##OP>::handle, Core::OPINDEX_F80##OP, SupportsPreserveAllABI}; \
return true; \
}
#define COMMON_BINARY_X87_OP(OP) \
case IR::OP_F80##OP: { \
*Info = {FABI_F80_I16_F80_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80##OP>::handle, Core::OPINDEX_F80##OP, SupportsPreserveAllABI}; \
return true; \
}
#define COMMON_F64_OP(OP) \
case IR::OP_F64##OP: { \
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F64##OP>::handle, Core::OPINDEX_F64##OP); \
return true; \
}
// Unary
COMMON_UNARY_X87_OP(ROUND)
@@ -242,20 +239,20 @@ bool InterpreterOps::GetFallbackHandler(IR::IROp_Header const *IROp, FallbackInf
COMMON_F64_OP(FPREM)
COMMON_F64_OP(SCALE)
// SSE4.2 Fallbacks
case IR::OP_VPCMPESTRX:
*Info = {FABI_I32_I64_I64_I128_I128_I16, (void*)&FEXCore::CPU::OpHandlers<IR::OP_VPCMPESTRX>::handle, Core::OPINDEX_VPCMPESTRX, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
case IR::OP_VPCMPISTRX:
*Info = {FABI_I32_I128_I128_I16, (void*)&FEXCore::CPU::OpHandlers<IR::OP_VPCMPISTRX>::handle, Core::OPINDEX_VPCMPISTRX, FEXCORE_HAS_PRESERVE_ALL_ATTR};
return true;
// SSE4.2 Fallbacks
case IR::OP_VPCMPESTRX:
*Info = {FABI_I32_I64_I64_I128_I128_I16, (void*)&FEXCore::CPU::OpHandlers<IR::OP_VPCMPESTRX>::handle, Core::OPINDEX_VPCMPESTRX,
SupportsPreserveAllABI};
return true;
case IR::OP_VPCMPISTRX:
*Info = {FABI_I32_I128_I128_I16, (void*)&FEXCore::CPU::OpHandlers<IR::OP_VPCMPISTRX>::handle, Core::OPINDEX_VPCMPISTRX, SupportsPreserveAllABI};
return true;
default:
break;
default: break;
}
return false;
}
}
} // namespace FEXCore::CPU
@@ -15,9 +15,9 @@ namespace FEXCore::CPU {
template<>
struct OpHandlers<IR::OP_VPCMPESTRX> {
enum class AggregationOp {
EqualAny = 0b00,
Ranges = 0b01,
EqualEach = 0b10,
EqualAny = 0b00,
Ranges = 0b01,
EqualEach = 0b10,
EqualOrdered = 0b11,
};
@@ -35,8 +35,7 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
NegativeMasked,
};
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t handle(uint64_t RAX, uint64_t RDX, __uint128_t lhs, __uint128_t rhs, uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t handle(uint64_t RAX, uint64_t RDX, __uint128_t lhs, __uint128_t rhs, uint16_t control) {
// Subtract by 1 in order to make validity limits 0-based
const auto valid_lhs = GetExplicitLength(RAX, control) - 1;
const auto valid_rhs = GetExplicitLength(RDX, control) - 1;
@@ -45,33 +44,31 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
}
// Main PCMPXSTRX algorithm body. Allows for reuse with both implicit and explicit length variants.
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t MainBody(const __uint128_t& lhs, int valid_lhs, const __uint128_t& rhs, int valid_rhs, uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t MainBody(const __uint128_t& lhs, int valid_lhs, const __uint128_t& rhs, int valid_rhs, uint16_t control) {
const uint32_t aggregation = PerformAggregation(lhs, valid_lhs, rhs, valid_rhs, control);
const int32_t upper_limit = (16 >> (control & 1)) - 1;
// Bits are arranged as:
// Bit #: 3 2 1 0
// [OF | CF | SF | ZF]
// [SF | ZF | CF | OF]
uint32_t flags = 0;
flags |= (valid_rhs < upper_limit) ? 0b01 : 0b00;
flags |= (valid_lhs < upper_limit) ? 0b10 : 0b00;
flags |= (valid_rhs < upper_limit) ? 0b0100 : 0b0000;
flags |= (valid_lhs < upper_limit) ? 0b1000 : 0b0000;
const uint32_t result = HandlePolarity(aggregation, control, upper_limit, valid_rhs);
if (result != 0) {
flags |= 0b0100;
flags |= 0b0010;
}
if ((result & 1) != 0) {
flags |= 0b1000;
flags |= 0b0001;
}
// We tack the flags on top of the result to avoid needing to handle
// multiple return values in the JITs.
return result | (flags << 16);
// We track the flags in the usual NZCV bit position so we can msr them
// later. Avoids handling flags natively in JIT.
return result | (flags << 28);
}
FEXCORE_PRESERVE_ALL_ATTR
static int32_t GetExplicitLength(uint64_t reg, uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static int32_t GetExplicitLength(uint64_t reg, uint16_t control) {
// Bit 8 controls whether or not the reg value is 64-bit or 32-bit.
int64_t value = 0;
if (((control >> 8) & 1) != 0) {
@@ -94,62 +91,50 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
return std::abs(static_cast<int>(value));
}
FEXCORE_PRESERVE_ALL_ATTR
static int32_t GetElement(const __uint128_t& vec, int32_t index, uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static int32_t GetElement(const __uint128_t& vec, int32_t index, uint16_t control) {
const auto* vec_ptr = reinterpret_cast<const uint8_t*>(&vec);
// Control bits [1:0] define the data type being dealt with.
switch (static_cast<SourceData>(control & 0b11)) {
case SourceData::U8:
return static_cast<int32_t>(vec_ptr[index]);
case SourceData::U8: return static_cast<int32_t>(vec_ptr[index]);
case SourceData::U16: {
uint16_t value{};
uint16_t value {};
std::memcpy(&value, vec_ptr + (sizeof(uint16_t) * static_cast<size_t>(index)), sizeof(value));
return value;
}
case SourceData::S8:
return static_cast<int8_t>(vec_ptr[index]);
case SourceData::S8: return static_cast<int8_t>(vec_ptr[index]);
case SourceData::S16:
default: {
int16_t value{};
int16_t value {};
std::memcpy(&value, vec_ptr + (sizeof(int16_t) * static_cast<size_t>(index)), sizeof(value));
return value;
}
}
}
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t PerformAggregation(const __uint128_t& lhs, int32_t valid_lhs,
const __uint128_t& rhs, int32_t valid_rhs,
uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t
PerformAggregation(const __uint128_t& lhs, int32_t valid_lhs, const __uint128_t& rhs, int32_t valid_rhs, uint16_t control) {
switch (static_cast<AggregationOp>((control >> 2) & 0b11)) {
case AggregationOp::EqualAny:
return HandleEqualAny(lhs, valid_lhs, rhs, valid_rhs, control);
case AggregationOp::Ranges:
return HandleRanges(lhs, valid_lhs, rhs, valid_rhs, control);
case AggregationOp::EqualEach:
return HandleEqualEach(lhs, valid_lhs, rhs, valid_rhs, control);
case AggregationOp::EqualAny: return HandleEqualAny(lhs, valid_lhs, rhs, valid_rhs, control);
case AggregationOp::Ranges: return HandleRanges(lhs, valid_lhs, rhs, valid_rhs, control);
case AggregationOp::EqualEach: return HandleEqualEach(lhs, valid_lhs, rhs, valid_rhs, control);
case AggregationOp::EqualOrdered:
default:
return HandleEqualOrdered(lhs, valid_lhs, rhs, valid_rhs, control);
default: return HandleEqualOrdered(lhs, valid_lhs, rhs, valid_rhs, control);
}
}
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t HandlePolarity(uint32_t value, uint16_t control, int upper_limit, int valid_rhs) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t HandlePolarity(uint32_t value, uint16_t control, int upper_limit, int valid_rhs) {
switch (static_cast<Polarity>((control >> 4) & 0b11)) {
case Polarity::Negative:
return value ^ ((2U << upper_limit) - 1);
case Polarity::NegativeMasked:
return value ^ ((1U << (valid_rhs + 1)) - 1);
case Polarity::Positive:
case Polarity::PositiveMasked:
default:
// Both positive masking and positive polarity are documented
// as both being equivalent to "IntRes2 = IntRes1", where IntRes1
// is our 'value' parameter, so we don't need to do anything in
// these cases except return the same value.
return value;
case Polarity::Negative: return value ^ ((2U << upper_limit) - 1);
case Polarity::NegativeMasked: return value ^ ((1U << (valid_rhs + 1)) - 1);
case Polarity::Positive:
case Polarity::PositiveMasked:
default:
// Both positive masking and positive polarity are documented
// as both being equivalent to "IntRes2 = IntRes1", where IntRes1
// is our 'value' parameter, so we don't need to do anything in
// these cases except return the same value.
return value;
}
}
@@ -175,10 +160,8 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
// │
// 'c' match ────────┘
//
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t HandleEqualAny(const __uint128_t& lhs, int32_t valid_lhs,
const __uint128_t& rhs, int32_t valid_rhs,
uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t
HandleEqualAny(const __uint128_t& lhs, int32_t valid_lhs, const __uint128_t& rhs, int32_t valid_rhs, uint16_t control) {
uint32_t result = 0;
for (int j = valid_rhs; j >= 0; j--) {
@@ -222,10 +205,8 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
// │
// 'Z' >= 'z' && 'A' <= 'z' ──────────┘
//
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t HandleRanges(const __uint128_t& lhs, int32_t valid_lhs,
const __uint128_t& rhs, int32_t valid_rhs,
uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t
HandleRanges(const __uint128_t& lhs, int32_t valid_lhs, const __uint128_t& rhs, int32_t valid_rhs, uint16_t control) {
uint32_t result = 0;
for (int j = valid_rhs; j >= 0; j--) {
@@ -275,10 +256,8 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
// │
// 'a' == 'a' ──────────┘
//
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t HandleEqualEach(const __uint128_t& lhs, int32_t valid_lhs,
const __uint128_t& rhs, int32_t valid_rhs,
uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t
HandleEqualEach(const __uint128_t& lhs, int32_t valid_lhs, const __uint128_t& rhs, int32_t valid_rhs, uint16_t control) {
const auto upper_limit = (16 >> (control & 1)) - 1;
const auto max_valid = std::max(valid_lhs, valid_rhs);
const auto min_valid = std::min(valid_lhs, valid_rhs);
@@ -330,10 +309,8 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
// │
// At index 0 ──────────┘
//
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t HandleEqualOrdered(const __uint128_t& lhs, int32_t valid_lhs,
const __uint128_t& rhs, int32_t valid_rhs,
uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t
HandleEqualOrdered(const __uint128_t& lhs, int32_t valid_lhs, const __uint128_t& rhs, int32_t valid_rhs, uint16_t control) {
const auto upper_limit = (16 >> (control & 1)) - 1;
// Edge case!
@@ -345,8 +322,7 @@ struct OpHandlers<IR::OP_VPCMPESTRX> {
}
uint32_t result = 0;
const int initial = valid_rhs == upper_limit ? valid_rhs
: valid_rhs - valid_lhs;
const int initial = valid_rhs == upper_limit ? valid_rhs : valid_rhs - valid_lhs;
for (int j = initial; j >= 0; j--) {
result <<= 1;
@@ -379,8 +355,7 @@ struct OpHandlers<IR::OP_VPCMPISTRX> {
// to be the max length possible for the given character size specified
// in the control flags (16 characters for 8-bit, and 8 characters for 16-bit).
//
FEXCORE_PRESERVE_ALL_ATTR
static uint32_t handle(__uint128_t lhs, __uint128_t rhs, uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static uint32_t handle(__uint128_t lhs, __uint128_t rhs, uint16_t control) {
// Subtract by 1 in order to make validity limits 0-based
const auto valid_lhs = GetImplicitLength(lhs, control) - 1;
const auto valid_rhs = GetImplicitLength(rhs, control) - 1;
@@ -388,8 +363,7 @@ struct OpHandlers<IR::OP_VPCMPISTRX> {
return OpHandlers<IR::OP_VPCMPESTRX>::MainBody(lhs, valid_lhs, rhs, valid_rhs, control);
}
FEXCORE_PRESERVE_ALL_ATTR
static int32_t GetImplicitLength(const __uint128_t& data, uint16_t control) {
FEXCORE_PRESERVE_ALL_ATTR static int32_t GetImplicitLength(const __uint128_t& data, uint16_t control) {
const auto* data_u8 = reinterpret_cast<const uint8_t*>(&data);
const auto is_using_words = (control & 1) != 0;
@@ -399,7 +373,7 @@ struct OpHandlers<IR::OP_VPCMPISTRX> {
const auto get_word = [data_u8](int32_t index) {
const auto* src = data_u8 + (index * sizeof(uint16_t));
uint16_t element{};
uint16_t element {};
std::memcpy(&element, src, sizeof(uint16_t));
return element;
};
@@ -7,44 +7,43 @@
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/IR/IR.h>
#include <FEXCore/IR/IntrusiveIRList.h>
namespace FEXCore::IR {
class IRListView;
struct IROp_Header;
}
class IRListView;
struct IROp_Header;
} // namespace FEXCore::IR
namespace FEXCore::CPU {
enum FallbackABI {
FABI_UNKNOWN,
FABI_F80_I16_F32,
FABI_F80_I16_F64,
FABI_F80_I16_I16,
FABI_F80_I16_I32,
FABI_F32_I16_F80,
FABI_F64_I16_F80,
FABI_F64_I16_F64,
FABI_F64_I16_F64_F64,
FABI_I16_I16_F80,
FABI_I32_I16_F80,
FABI_I64_I16_F80,
FABI_I64_I16_F80_F80,
FABI_F80_I16_F80,
FABI_F80_I16_F80_F80,
FABI_I32_I64_I64_I128_I128_I16,
FABI_I32_I128_I128_I16,
};
enum FallbackABI {
FABI_UNKNOWN,
FABI_F80_I16_F32,
FABI_F80_I16_F64,
FABI_F80_I16_I16,
FABI_F80_I16_I32,
FABI_F32_I16_F80,
FABI_F64_I16_F80,
FABI_F64_I16_F64,
FABI_F64_I16_F64_F64,
FABI_I16_I16_F80,
FABI_I32_I16_F80,
FABI_I64_I16_F80,
FABI_I64_I16_F80_F80,
FABI_F80_I16_F80,
FABI_F80_I16_F80_F80,
FABI_I32_I64_I64_I128_I128_I16,
FABI_I32_I128_I128_I16,
};
struct FallbackInfo {
FallbackABI ABI;
void *fn;
FEXCore::Core::FallbackHandlerIndex HandlerIndex;
bool SupportsPreserveAllABI;
};
struct FallbackInfo {
FallbackABI ABI;
void* fn;
FEXCore::Core::FallbackHandlerIndex HandlerIndex;
bool SupportsPreserveAllABI;
};
class InterpreterOps {
public:
static void FillFallbackIndexPointers(uint64_t *Info);
static bool GetFallbackHandler(IR::IROp_Header const *IROp, FallbackInfo *Info);
};
class InterpreterOps {
public:
static void FillFallbackIndexPointers(uint64_t* Info);
static bool GetFallbackHandler(bool SupportsPreserveAllABI, const IR::IROp_Header* IROp, FallbackInfo* Info);
};
} // namespace FEXCore::CPU
File diff suppressed because it is too large. Load diff
@@ -13,21 +13,19 @@ namespace FEXCore::CPU {
uint64_t Arm64JITCore::GetNamedSymbolLiteral(FEXCore::CPU::RelocNamedSymbolLiteral::NamedSymbol Op) {
switch (Op) {
case FEXCore::CPU::RelocNamedSymbolLiteral::NamedSymbol::SYMBOL_LITERAL_EXITFUNCTION_LINKER:
return ThreadState->CurrentFrame->Pointers.Common.ExitFunctionLinker;
break;
default:
ERROR_AND_DIE_FMT("Unknown named symbol literal: {}", static_cast<uint32_t>(Op));
case FEXCore::CPU::RelocNamedSymbolLiteral::NamedSymbol::SYMBOL_LITERAL_EXITFUNCTION_LINKER:
return ThreadState->CurrentFrame->Pointers.Common.ExitFunctionLinker;
break;
default: ERROR_AND_DIE_FMT("Unknown named symbol literal: {}", static_cast<uint32_t>(Op)); break;
}
return ~0ULL;
}
void Arm64JITCore::InsertNamedThunkRelocation(ARMEmitter::Register Reg, const IR::SHA256Sum &Sum) {
Relocation MoveABI{};
void Arm64JITCore::InsertNamedThunkRelocation(ARMEmitter::Register Reg, const IR::SHA256Sum& Sum) {
Relocation MoveABI {};
MoveABI.NamedThunkMove.Header.Type = FEXCore::CPU::RelocationTypes::RELOC_NAMED_THUNK_MOVE;
// Offset is the offset from the entrypoint of the block
auto CurrentCursor = GetCursorAddress<uint8_t *>();
auto CurrentCursor = GetCursorAddress<uint8_t*>();
MoveABI.NamedThunkMove.Offset = CurrentCursor - CodeData.BlockBegin;
MoveABI.NamedThunkMove.Symbol = Sum;
MoveABI.NamedThunkMove.RegisterIndex = Reg.Idx();
@@ -43,22 +41,25 @@ Arm64JITCore::NamedSymbolLiteralPair Arm64JITCore::InsertNamedSymbolLiteral(FEXC
Arm64JITCore::NamedSymbolLiteralPair Lit {
.Lit = Pointer,
.MoveABI = {
.NamedSymbolLiteral = {
.Header = {
.Type = FEXCore::CPU::RelocationTypes::RELOC_NAMED_SYMBOL_LITERAL,
},
.Symbol = Op,
.Offset = 0,
.MoveABI =
{
.NamedSymbolLiteral =
{
.Header =
{
.Type = FEXCore::CPU::RelocationTypes::RELOC_NAMED_SYMBOL_LITERAL,
},
.Symbol = Op,
.Offset = 0,
},
},
},
};
return Lit;
}
void Arm64JITCore::PlaceNamedSymbolLiteral(NamedSymbolLiteralPair &Lit) {
void Arm64JITCore::PlaceNamedSymbolLiteral(NamedSymbolLiteralPair& Lit) {
// Offset is the offset from the entrypoint of the block
auto CurrentCursor = GetCursorAddress<uint8_t *>();
auto CurrentCursor = GetCursorAddress<uint8_t*>();
Lit.MoveABI.NamedSymbolLiteral.Offset = CurrentCursor - CodeData.BlockBegin;
Bind(&Lit.Loc);
@@ -67,10 +68,10 @@ void Arm64JITCore::PlaceNamedSymbolLiteral(NamedSymbolLiteralPair &Lit) {
}
void Arm64JITCore::InsertGuestRIPMove(ARMEmitter::Register Reg, uint64_t Constant) {
Relocation MoveABI{};
Relocation MoveABI {};
MoveABI.GuestRIPMove.Header.Type = FEXCore::CPU::RelocationTypes::RELOC_GUEST_RIP_MOVE;
// Offset is the offset from the entrypoint of the block
auto CurrentCursor = GetCursorAddress<uint8_t *>();
auto CurrentCursor = GetCursorAddress<uint8_t*>();
MoveABI.GuestRIPMove.Offset = CurrentCursor - CodeData.BlockBegin;
MoveABI.GuestRIPMove.GuestRIP = Constant;
MoveABI.GuestRIPMove.RegisterIndex = Reg.Idx();
@@ -79,54 +80,54 @@ void Arm64JITCore::InsertGuestRIPMove(ARMEmitter::Register Reg, uint64_t Constan
Relocations.emplace_back(MoveABI);
}
bool Arm64JITCore::ApplyRelocations(uint64_t GuestEntry, uint64_t CodeEntry, uint64_t CursorEntry, size_t NumRelocations, const char* EntryRelocations) {
size_t DataIndex{};
bool Arm64JITCore::ApplyRelocations(uint64_t GuestEntry, uint64_t CodeEntry, uint64_t CursorEntry, size_t NumRelocations,
const char* EntryRelocations) {
size_t DataIndex {};
for (size_t j = 0; j < NumRelocations; ++j) {
const FEXCore::CPU::Relocation *Reloc = reinterpret_cast<const FEXCore::CPU::Relocation *>(&EntryRelocations[DataIndex]);
const FEXCore::CPU::Relocation* Reloc = reinterpret_cast<const FEXCore::CPU::Relocation*>(&EntryRelocations[DataIndex]);
LOGMAN_THROW_AA_FMT((DataIndex % alignof(Relocation)) == 0, "Alignment of relocation wasn't adhered to");
switch (Reloc->Header.Type) {
case FEXCore::CPU::RelocationTypes::RELOC_NAMED_SYMBOL_LITERAL: {
uint64_t Pointer = GetNamedSymbolLiteral(Reloc->NamedSymbolLiteral.Symbol);
// Relocation occurs at the cursorEntry + offset relative to that cursor
SetCursorOffset(CursorEntry + Reloc->NamedSymbolLiteral.Offset);
case FEXCore::CPU::RelocationTypes::RELOC_NAMED_SYMBOL_LITERAL: {
uint64_t Pointer = GetNamedSymbolLiteral(Reloc->NamedSymbolLiteral.Symbol);
// Relocation occurs at the cursorEntry + offset relative to that cursor
SetCursorOffset(CursorEntry + Reloc->NamedSymbolLiteral.Offset);
// Generate a literal so we can place it
dc64(Pointer);
// Generate a literal so we can place it
dc64(Pointer);
DataIndex += sizeof(Reloc->NamedSymbolLiteral);
break;
DataIndex += sizeof(Reloc->NamedSymbolLiteral);
break;
}
case FEXCore::CPU::RelocationTypes::RELOC_NAMED_THUNK_MOVE: {
uint64_t Pointer = reinterpret_cast<uint64_t>(EmitterCTX->ThunkHandler->LookupThunk(Reloc->NamedThunkMove.Symbol));
if (Pointer == ~0ULL) {
return false;
}
case FEXCore::CPU::RelocationTypes::RELOC_NAMED_THUNK_MOVE: {
uint64_t Pointer = reinterpret_cast<uint64_t>(EmitterCTX->ThunkHandler->LookupThunk(Reloc->NamedThunkMove.Symbol));
if (Pointer == ~0ULL) {
return false;
}
// Relocation occurs at the cursorEntry + offset relative to that cursor.
SetCursorOffset(CursorEntry + Reloc->NamedThunkMove.Offset);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Register(Reloc->NamedThunkMove.RegisterIndex), Pointer, true);
DataIndex += sizeof(Reloc->NamedThunkMove);
break;
// Relocation occurs at the cursorEntry + offset relative to that cursor.
SetCursorOffset(CursorEntry + Reloc->NamedThunkMove.Offset);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Register(Reloc->NamedThunkMove.RegisterIndex), Pointer, true);
DataIndex += sizeof(Reloc->NamedThunkMove);
break;
}
case FEXCore::CPU::RelocationTypes::RELOC_GUEST_RIP_MOVE: {
// XXX: Reenable once the JIT Object Cache is upstream
// XXX: Should spin the relocation list, create a list of guest RIP moves, and ask for them all once, reduces lock contention.
uint64_t Pointer = ~0ULL; // EmitterCTX->JITObjectCache->FindRelocatedRIP(Reloc->GuestRIPMove.GuestRIP);
if (Pointer == ~0ULL) {
return false;
}
case FEXCore::CPU::RelocationTypes::RELOC_GUEST_RIP_MOVE: {
// XXX: Reenable once the JIT Object Cache is upstream
// XXX: Should spin the relocation list, create a list of guest RIP moves, and ask for them all once, reduces lock contention.
uint64_t Pointer = ~0ULL; // EmitterCTX->JITObjectCache->FindRelocatedRIP(Reloc->GuestRIPMove.GuestRIP);
if (Pointer == ~0ULL) {
return false;
}
// Relocation occurs at the cursorEntry + offset relative to that cursor.
SetCursorOffset(CursorEntry + Reloc->GuestRIPMove.Offset);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Register(Reloc->GuestRIPMove.RegisterIndex), Pointer, true);
DataIndex += sizeof(Reloc->GuestRIPMove);
break;
}
// Relocation occurs at the cursorEntry + offset relative to that cursor.
SetCursorOffset(CursorEntry + Reloc->GuestRIPMove.Offset);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Register(Reloc->GuestRIPMove.RegisterIndex), Pointer, true);
DataIndex += sizeof(Reloc->GuestRIPMove);
break;
}
}
}
return true;
}
}
} // namespace FEXCore::CPU
@@ -6,12 +6,11 @@ $end_info$
*/
#include "Interface/Context/Context.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(CASPair) {
auto Op = IROp->C<IR::IROp_CASPair>();
LOGMAN_THROW_AA_FMT(IROp->ElementSize == 4 || IROp->ElementSize == 8, "Wrong element size");
@@ -29,13 +28,20 @@ DEF_OP(CASPair) {
caspal(EmitSize, TMP3, TMP4, Desired.first, Desired.second, MemSrc);
mov(EmitSize, Dst.first, TMP3.R());
mov(EmitSize, Dst.second, TMP4.R());
}
else {
} else {
// Save NZCV so we don't have to mark this op as clobbering NZCV (the
// SupportsAtomics does not clobber atomics and this !SupportsAtomics path
// is so slow it's not worth the complexity of splitting the IR op.). We
// clobber NZCV inside the hot loop and we can't replace cmp/ccmp/b.ne with
// something NZCV-preserving without requiring an extra instruction.
mrs(TMP1, ARMEmitter::SystemRegister::NZCV);
ARMEmitter::BackwardLabel LoopTop;
ARMEmitter::ForwardLabel LoopNotExpected;
ARMEmitter::ForwardLabel LoopExpected;
ARMEmitter::SingleUseForwardLabel LoopNotExpected;
ARMEmitter::SingleUseForwardLabel LoopExpected;
Bind(&LoopTop);
// This instruction sequence must be synced with HandleCASPAL_Armv8.
ldaxp(EmitSize, TMP2, TMP3, MemSrc);
cmp(EmitSize, TMP2, Expected.first);
ccmp(EmitSize, TMP3, Expected.second, ARMEmitter::StatusFlags::None, ARMEmitter::Condition::CC_EQ);
@@ -47,20 +53,23 @@ DEF_OP(CASPair) {
b(&LoopExpected);
Bind(&LoopNotExpected);
mov(EmitSize, Dst.first, TMP2.R());
mov(EmitSize, Dst.second, TMP3.R());
// exclusive monitor needs to be cleared here
// Might have hit the case where ldaxr was hit but stlxr wasn't
clrex();
Bind(&LoopNotExpected);
mov(EmitSize, Dst.first, TMP2.R());
mov(EmitSize, Dst.second, TMP3.R());
// exclusive monitor needs to be cleared here
// Might have hit the case where ldaxr was hit but stlxr wasn't
clrex();
Bind(&LoopExpected);
// Restore
msr(ARMEmitter::SystemRegister::NZCV, TMP1);
}
}
DEF_OP(CAS) {
auto Op = IROp->C<IR::IROp_CAS>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
// DataSrc = *Src1
// if (DataSrc == Src3) { *Src1 == Src2; } Src2 = DataSrc
// This will write to memory! Careful!
@@ -69,30 +78,21 @@ DEF_OP(CAS) {
auto Desired = GetReg(Op->Desired.ID());
auto MemSrc = GetReg(Op->Addr.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
mov(EmitSize, TMP2, Expected);
casal(SubEmitSize, TMP2, Desired, MemSrc);
mov(EmitSize, GetReg(Node), TMP2.R());
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
ARMEmitter::ForwardLabel LoopNotExpected;
ARMEmitter::ForwardLabel LoopExpected;
ARMEmitter::SingleUseForwardLabel LoopNotExpected;
ARMEmitter::SingleUseForwardLabel LoopExpected;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
if (OpSize == 1) {
if (IROp->Size == 1) {
cmp(EmitSize, TMP2, Expected, ARMEmitter::ExtendedType::UXTB, 0);
}
else if (OpSize == 2) {
} else if (IROp->Size == 2) {
cmp(EmitSize, TMP2, Expected, ARMEmitter::ExtendedType::UXTH, 0);
}
else {
} else {
cmp(EmitSize, TMP2, Expected);
}
b(ARMEmitter::Condition::CC_NE, &LoopNotExpected);
@@ -101,33 +101,26 @@ DEF_OP(CAS) {
mov(EmitSize, GetReg(Node), Expected);
b(&LoopExpected);
Bind(&LoopNotExpected);
mov(EmitSize, GetReg(Node), TMP2.R());
// exclusive monitor needs to be cleared here
// Might have hit the case where ldaxr was hit but stlxr wasn't
clrex();
Bind(&LoopNotExpected);
mov(EmitSize, GetReg(Node), TMP2.R());
// exclusive monitor needs to be cleared here
// Might have hit the case where ldaxr was hit but stlxr wasn't
clrex();
Bind(&LoopExpected);
}
}
DEF_OP(AtomicAdd) {
auto Op = IROp->C<IR::IROp_AtomicAdd>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
staddl(SubEmitSize, Src, MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -139,23 +132,16 @@ DEF_OP(AtomicAdd) {
DEF_OP(AtomicSub) {
auto Op = IROp->C<IR::IROp_AtomicSub>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
neg(EmitSize, TMP2, Src);
staddl(SubEmitSize, TMP2, MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -167,23 +153,16 @@ DEF_OP(AtomicSub) {
DEF_OP(AtomicAnd) {
auto Op = IROp->C<IR::IROp_AtomicAnd>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
mvn(EmitSize, TMP2, Src);
stclrl(SubEmitSize, TMP2, MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -193,24 +172,37 @@ DEF_OP(AtomicAnd) {
}
}
DEF_OP(AtomicOr) {
auto Op = IROp->C<IR::IROp_AtomicOr>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
DEF_OP(AtomicCLR) {
auto Op = IROp->C<IR::IROp_AtomicCLR>();
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
stclrl(SubEmitSize, Src, MemSrc);
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
bic(EmitSize, TMP2, TMP2, Src);
stlxr(SubEmitSize, TMP2, TMP2, MemSrc);
cbnz(EmitSize, TMP2, &LoopTop);
}
}
DEF_OP(AtomicOr) {
auto Op = IROp->C<IR::IROp_AtomicOr>();
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
if (CTX->HostFeatures.SupportsAtomics) {
stsetl(SubEmitSize, Src, MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -222,22 +214,15 @@ DEF_OP(AtomicOr) {
DEF_OP(AtomicXor) {
auto Op = IROp->C<IR::IROp_AtomicXor>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
steorl(SubEmitSize, Src, MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -247,6 +232,21 @@ DEF_OP(AtomicXor) {
}
}
DEF_OP(AtomicNeg) {
auto Op = IROp->C<IR::IROp_AtomicNeg>();
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
neg(EmitSize, TMP3, TMP2);
stlxr(SubEmitSize, TMP4, TMP3, MemSrc);
cbnz(EmitSize, TMP4, &LoopTop);
}
DEF_OP(AtomicSwap) {
auto Op = IROp->C<IR::IROp_AtomicSwap>();
uint8_t OpSize = IROp->Size;
@@ -255,17 +255,16 @@ DEF_OP(AtomicSwap) {
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit :
ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
mov(EmitSize, TMP2, Src);
ldswpal(SubEmitSize, TMP2, GetReg(Node), MemSrc);
}
else {
ldswpal(SubEmitSize, Src, GetReg(Node), MemSrc);
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -277,22 +276,15 @@ DEF_OP(AtomicSwap) {
DEF_OP(AtomicFetchAdd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAdd>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
ldaddal(SubEmitSize, Src, GetReg(Node), MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -305,23 +297,16 @@ DEF_OP(AtomicFetchAdd) {
DEF_OP(AtomicFetchSub) {
auto Op = IROp->C<IR::IROp_AtomicFetchSub>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
neg(EmitSize, TMP2, Src);
ldaddal(SubEmitSize, TMP2, GetReg(Node), MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -334,23 +319,16 @@ DEF_OP(AtomicFetchSub) {
DEF_OP(AtomicFetchAnd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAnd>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
mvn(EmitSize, TMP2, Src);
ldclral(SubEmitSize, TMP2, GetReg(Node), MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -363,22 +341,15 @@ DEF_OP(AtomicFetchAnd) {
DEF_OP(AtomicFetchCLR) {
auto Op = IROp->C<IR::IROp_AtomicFetchCLR>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
ldclral(SubEmitSize, Src, GetReg(Node), MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -391,22 +362,15 @@ DEF_OP(AtomicFetchCLR) {
DEF_OP(AtomicFetchOr) {
auto Op = IROp->C<IR::IROp_AtomicFetchOr>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
ldsetal(SubEmitSize, Src, GetReg(Node), MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -419,22 +383,15 @@ DEF_OP(AtomicFetchOr) {
DEF_OP(AtomicFetchXor) {
auto Op = IROp->C<IR::IROp_AtomicFetchXor>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
ldeoral(SubEmitSize, Src, GetReg(Node), MemSrc);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -447,17 +404,11 @@ DEF_OP(AtomicFetchXor) {
DEF_OP(AtomicFetchNeg) {
auto Op = IROp->C<IR::IROp_AtomicFetchNeg>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = ConvertSubRegSize8(IROp->Size);
auto MemSrc = GetReg(Op->Addr.ID());
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
@@ -480,8 +431,7 @@ DEF_OP(TelemetrySetValue) {
if (CTX->HostFeatures.SupportsAtomics) {
stsetl(ARMEmitter::SubRegSize::i64Bit, TMP1, TMP2);
}
else {
} else {
ARMEmitter::BackwardLabel LoopTop;
Bind(&LoopTop);
ldaxr(ARMEmitter::SubRegSize::i64Bit, TMP3, TMP2);
@@ -493,5 +443,4 @@ DEF_OP(TelemetrySetValue) {
}
#undef DEF_OP
}
} // namespace FEXCore::CPU
@@ -7,19 +7,18 @@ $end_info$
#include "Interface/Context/Context.h"
#include "FEXCore/IR/IR.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/LookupCache.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/InternalThreadState.h"
#include <FEXCore/Core/X86Enums.h>
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXCore/HLE/SyscallHandler.h>
#include <FEXCore/Utils/MathUtils.h>
#include <Interface/HLE/Thunks/Thunks.h>
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(CallbackReturn) {
// spill back to CTX
@@ -53,32 +52,40 @@ DEF_OP(ExitFunction) {
uint64_t NewRIP;
if (IsInlineConstant(Op->NewRIP, &NewRIP) || IsInlineEntrypointOffset(Op->NewRIP, &NewRIP)) {
ARMEmitter::ForwardLabel l_BranchHost;
ARMEmitter::ForwardLabel l_BranchGuest;
ldr(ARMEmitter::XReg::x0, &l_BranchHost);
blr(ARMEmitter::Reg::r0);
Bind(&l_BranchHost);
dc64(ThreadState->CurrentFrame->Pointers.Common.ExitFunctionLinker);
Bind(&l_BranchGuest);
dc64(NewRIP);
#ifdef _M_ARM_64EC
if (RtlIsEcCode(NewRIP)) {
add(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::rsp, StaticRegisters[X86State::REG_RSP], 0);
LoadConstant(ARMEmitter::Size::i64Bit, EC_CALL_CHECKER_PC_REG, NewRIP);
ldr(TMP2, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionEC));
br(TMP2);
} else {
#endif
ARMEmitter::SingleUseForwardLabel l_BranchHost;
ldr(TMP1, &l_BranchHost);
blr(TMP1);
Bind(&l_BranchHost);
dc64(ThreadState->CurrentFrame->Pointers.Common.ExitFunctionLinker);
dc64(NewRIP);
#ifdef _M_ARM_64EC
}
#endif
} else {
ARMEmitter::ForwardLabel FullLookup;
ARMEmitter::SingleUseForwardLabel FullLookup;
auto RipReg = GetReg(Op->NewRIP.ID());
// L1 Cache
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.L1Pointer));
ldr(TMP1, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.L1Pointer));
and_(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, RipReg, LookupCache::L1_ENTRIES_MASK);
add(ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, ARMEmitter::XReg::x3, ARMEmitter::ShiftType::LSL, 4);
and_(ARMEmitter::Size::i64Bit, TMP4, RipReg, LookupCache::L1_ENTRIES_MASK);
add(TMP1, TMP1, TMP4, ARMEmitter::ShiftType::LSL, 4);
ldp<ARMEmitter::IndexType::OFFSET>(ARMEmitter::XReg::x1, ARMEmitter::XReg::x0, ARMEmitter::Reg::r0, 0);
cmp(ARMEmitter::XReg::x0, RipReg.X());
b(ARMEmitter::Condition::CC_NE, &FullLookup);
br(ARMEmitter::Reg::r1);
// Note: sub+cbnz used over cmp+br to preserve flags.
ldp<ARMEmitter::IndexType::OFFSET>(TMP2, TMP1, TMP1, 0);
sub(TMP1, TMP1, RipReg.X());
cbnz(ARMEmitter::Size::i64Bit, TMP1, &FullLookup);
br(TMP2);
Bind(&FullLookup);
ldr(TMP1, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.DispatcherLoopTop));
@@ -94,77 +101,31 @@ DEF_OP(Jump) {
PendingTargetLabel = &JumpTargets.try_emplace(Target).first->second;
}
static ARMEmitter::Condition MapBranchCC(IR::CondClassType Cond) {
switch (Cond.Val) {
case FEXCore::IR::COND_ANDZ:
case FEXCore::IR::COND_EQ: return ARMEmitter::Condition::CC_EQ;
case FEXCore::IR::COND_ANDNZ:
case FEXCore::IR::COND_NEQ: return ARMEmitter::Condition::CC_NE;
case FEXCore::IR::COND_SGE: return ARMEmitter::Condition::CC_GE;
case FEXCore::IR::COND_SLT: return ARMEmitter::Condition::CC_LT;
case FEXCore::IR::COND_SGT: return ARMEmitter::Condition::CC_GT;
case FEXCore::IR::COND_SLE: return ARMEmitter::Condition::CC_LE;
case FEXCore::IR::COND_UGE: return ARMEmitter::Condition::CC_CS;
case FEXCore::IR::COND_ULT: return ARMEmitter::Condition::CC_CC;
case FEXCore::IR::COND_UGT: return ARMEmitter::Condition::CC_HI;
case FEXCore::IR::COND_ULE: return ARMEmitter::Condition::CC_LS;
case FEXCore::IR::COND_FLU: return ARMEmitter::Condition::CC_LT;
case FEXCore::IR::COND_FGE: return ARMEmitter::Condition::CC_GE;
case FEXCore::IR::COND_FLEU:return ARMEmitter::Condition::CC_LE;
case FEXCore::IR::COND_FGT: return ARMEmitter::Condition::CC_GT;
case FEXCore::IR::COND_FU: return ARMEmitter::Condition::CC_VS;
case FEXCore::IR::COND_FNU: return ARMEmitter::Condition::CC_VC;
case FEXCore::IR::COND_VS:
case FEXCore::IR::COND_VC:
case FEXCore::IR::COND_MI:
case FEXCore::IR::COND_PL:
default:
LOGMAN_MSG_A_FMT("Unsupported compare type");
return ARMEmitter::Condition::CC_NV;
}
}
DEF_OP(CondJump) {
auto Op = IROp->C<IR::IROp_CondJump>();
auto TrueTargetLabel = &JumpTargets.try_emplace(Op->TrueBlock.ID()).first->second;
uint64_t Const;
const bool isConst = IsInlineConstant(Op->Cmp2, &Const);
bool tests = Op->Cond == FEXCore::IR::COND_ANDZ ||
Op->Cond == FEXCore::IR::COND_ANDNZ;
const auto Size = Op->CompareSize == 4 ? ARMEmitter::Size::i32Bit : ARMEmitter::Size::i64Bit;
const auto SubSize = ARMEmitter::ToVectorSizePair(Op->CompareSize == 4 ? ARMEmitter::SubRegSize::i32Bit : ARMEmitter::SubRegSize::i64Bit);
if (isConst && Const == 0 && Op->Cond.Val == FEXCore::IR::COND_EQ) {
LOGMAN_THROW_A_FMT(IsGPR(Op->Cmp1.ID()), "CondJump: Expected GPR");
cbz(Size, GetReg(Op->Cmp1.ID()), TrueTargetLabel);
} else if (isConst && Const == 0 && Op->Cond.Val == FEXCore::IR::COND_NEQ) {
LOGMAN_THROW_A_FMT(IsGPR(Op->Cmp1.ID()), "CondJump: Expected GPR");
cbnz(Size, GetReg(Op->Cmp1.ID()), TrueTargetLabel);
if (Op->FromNZCV) {
b(MapCC(Op->Cond), TrueTargetLabel);
} else {
if (IsGPR(Op->Cmp1.ID())) {
if (tests) {
if (isConst) {
tst(Size, GetReg(Op->Cmp1.ID()), Const);
} else {
tst(Size, GetReg(Op->Cmp1.ID()), GetReg(Op->Cmp2.ID()));
}
} else {
if (isConst) {
cmp(Size, GetReg(Op->Cmp1.ID()), Const);
} else {
cmp(Size, GetReg(Op->Cmp1.ID()), GetReg(Op->Cmp2.ID()));
}
}
} else if (IsFPR(Op->Cmp1.ID())) {
fcmp(SubSize.Scalar, GetVReg(Op->Cmp1.ID()), GetVReg(Op->Cmp2.ID()));
[[maybe_unused]] uint64_t Const;
[[maybe_unused]] const bool isConst = IsInlineConstant(Op->Cmp2, &Const);
const auto Size = Op->CompareSize == 4 ? ARMEmitter::Size::i32Bit : ARMEmitter::Size::i64Bit;
LOGMAN_THROW_A_FMT(IsGPR(Op->Cmp1.ID()), "CondJump: Expected GPR");
LOGMAN_THROW_A_FMT(isConst && Const == 0, "CondJump: Expected 0 source");
LOGMAN_THROW_A_FMT(Op->Cond.Val == FEXCore::IR::COND_EQ || Op->Cond.Val == FEXCore::IR::COND_NEQ, "CondJump: Expected simple "
"condition");
if (Op->Cond.Val == FEXCore::IR::COND_EQ) {
cbz(Size, GetReg(Op->Cmp1.ID()), TrueTargetLabel);
} else {
LOGMAN_MSG_A_FMT("CondJump: Expected GPR or FPR");
cbnz(Size, GetReg(Op->Cmp1.ID()), TrueTargetLabel);
}
b(MapBranchCC(Op->Cond), TrueTargetLabel);
// TODO: Wire up tbz/tbnz
}
PendingTargetLabel = &JumpTargets.try_emplace(Op->FalseBlock.ID()).first->second;
@@ -200,7 +161,9 @@ DEF_OP(Syscall) {
uint64_t SPOffset = AlignUp(FEXCore::HLE::SyscallArguments::MAX_ARGS * 8, 16);
sub(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::rsp, ARMEmitter::Reg::rsp, SPOffset);
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS; ++i) {
if (Op->Header.Args[i].IsInvalid()) continue;
if (Op->Header.Args[i].IsInvalid()) {
continue;
}
str(GetReg(Op->Header.Args[i].ID()).X(), ARMEmitter::Reg::rsp, i * 8);
}
@@ -212,8 +175,7 @@ DEF_OP(Syscall) {
add(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r2, ARMEmitter::Reg::rsp, 0);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uint64_t, void*, void*, void*>(ARMEmitter::Reg::r3);
}
else {
} else {
blr(ARMEmitter::Reg::r3);
}
@@ -251,20 +213,19 @@ DEF_OP(InlineSyscall) {
// X6: Arg6 - Doesn't exist in x86-64 land. RA INTERSECT
// One argument is removed from the SyscallArguments::MAX_ARGS since the first argument was syscall number
const static std::array<ARMEmitter::XRegister, FEXCore::HLE::SyscallArguments::MAX_ARGS-1> RegArgs = {{
ARMEmitter::XReg::x0, ARMEmitter::XReg::x1, ARMEmitter::XReg::x2, ARMEmitter::XReg::x3, ARMEmitter::XReg::x4, ARMEmitter::XReg::x5
}};
const static std::array<ARMEmitter::XRegister, FEXCore::HLE::SyscallArguments::MAX_ARGS - 1> RegArgs = {
{ARMEmitter::XReg::x0, ARMEmitter::XReg::x1, ARMEmitter::XReg::x2, ARMEmitter::XReg::x3, ARMEmitter::XReg::x4, ARMEmitter::XReg::x5}};
bool Intersects{};
bool Intersects {};
// We always need to spill x8 since we can't know if it is live at this SSA location
uint32_t SpillMask = 1U << 8;
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS-1; ++i) {
if (Op->Header.Args[i].IsInvalid()) break;
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS - 1; ++i) {
if (Op->Header.Args[i].IsInvalid()) {
break;
}
auto Reg = GetReg(Op->Header.Args[i].ID());
if (Reg == ARMEmitter::Reg::r8 ||
Reg == ARMEmitter::Reg::r4 ||
Reg == ARMEmitter::Reg::r5) {
if (Reg == ARMEmitter::Reg::r8 || Reg == ARMEmitter::Reg::r4 || Reg == ARMEmitter::Reg::r5) {
SpillMask |= (1U << Reg.Idx());
Intersects = true;
@@ -288,8 +249,10 @@ DEF_OP(InlineSyscall) {
const auto EmitSize = CTX->Config.Is64BitMode() ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto EmitSubSize = CTX->Config.Is64BitMode() ? ARMEmitter::SubRegSize::i64Bit : ARMEmitter::SubRegSize::i32Bit;
if (Intersects) {
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS-1; ++i) {
if (Op->Header.Args[i].IsInvalid()) break;
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS - 1; ++i) {
if (Op->Header.Args[i].IsInvalid()) {
break;
}
auto Reg = GetReg(Op->Header.Args[i].ID());
// In the case of intersection with x4, x5, or x8 then these are currently SRA
@@ -297,21 +260,19 @@ DEF_OP(InlineSyscall) {
// Just load back from the context. Could be slightly smarter but this is fairly uncommon
if (Reg == ARMEmitter::Reg::r8) {
ldr(EmitSubSize, RegArgs[i].R(), STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[X86State::REG_RSP]));
}
else if (Reg == ARMEmitter::Reg::r4) {
} else if (Reg == ARMEmitter::Reg::r4) {
ldr(EmitSubSize, RegArgs[i].R(), STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[X86State::REG_RAX]));
}
else if (Reg == ARMEmitter::Reg::r5) {
} else if (Reg == ARMEmitter::Reg::r5) {
ldr(EmitSubSize, RegArgs[i].R(), STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[X86State::REG_RCX]));
}
else {
} else {
mov(EmitSize, RegArgs[i].R(), Reg);
}
}
}
else {
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS-1; ++i) {
if (Op->Header.Args[i].IsInvalid()) break;
} else {
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS - 1; ++i) {
if (Op->Header.Args[i].IsInvalid()) {
break;
}
mov(EmitSize, RegArgs[i].R(), GetReg(Op->Header.Args[i].ID()));
}
@@ -352,8 +313,7 @@ DEF_OP(Thunk) {
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r2, (uintptr_t)thunkFn);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<void, void*, void*>(ARMEmitter::Reg::r2);
}
else {
} else {
blr(ARMEmitter::Reg::r2);
}
@@ -364,70 +324,65 @@ DEF_OP(Thunk) {
DEF_OP(ValidateCode) {
auto Op = IROp->C<IR::IROp_ValidateCode>();
const auto *OldCode = (const uint8_t *)&Op->CodeOriginalLow;
const auto* OldCode = (const uint8_t*)&Op->CodeOriginalLow;
int len = Op->CodeLength;
int idx = 0;
LoadConstant(ARMEmitter::Size::i64Bit, GetReg(Node), 0);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, Entry + Op->Offset);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, 1);
LoadConstant(ARMEmitter::Size::i64Bit, TMP1, Entry + Op->Offset);
LoadConstant(ARMEmitter::Size::i64Bit, TMP2, 1);
const auto Dst = GetReg(Node);
while (len >= 8)
{
ldr(ARMEmitter::XReg::x2, ARMEmitter::Reg::r0, idx);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, *(const uint32_t *)(OldCode + idx));
cmp(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r2, ARMEmitter::Reg::r3);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, ARMEmitter::Reg::r1, ARMEmitter::Condition::CC_EQ);
while (len >= 8) {
ldr(ARMEmitter::XReg::x2, TMP1, idx);
LoadConstant(ARMEmitter::Size::i64Bit, TMP4, *(const uint32_t*)(OldCode + idx));
cmp(ARMEmitter::Size::i64Bit, TMP3, TMP4);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, TMP2, ARMEmitter::Condition::CC_EQ);
len -= 8;
idx += 8;
}
while (len >= 4)
{
ldr(ARMEmitter::WReg::w2, ARMEmitter::Reg::r0, idx);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, *(const uint32_t *)(OldCode + idx));
cmp(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r2, ARMEmitter::Reg::r3);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, ARMEmitter::Reg::r1, ARMEmitter::Condition::CC_EQ);
while (len >= 4) {
ldr(ARMEmitter::WReg::w2, TMP1, idx);
LoadConstant(ARMEmitter::Size::i64Bit, TMP4, *(const uint32_t*)(OldCode + idx));
cmp(ARMEmitter::Size::i32Bit, TMP3, TMP4);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, TMP2, ARMEmitter::Condition::CC_EQ);
len -= 4;
idx += 4;
}
while (len >= 2)
{
ldrh(ARMEmitter::Reg::r2, ARMEmitter::Reg::r0, idx);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, *(const uint16_t *)(OldCode + idx));
cmp(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r2, ARMEmitter::Reg::r3);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, ARMEmitter::Reg::r1, ARMEmitter::Condition::CC_EQ);
while (len >= 2) {
ldrh(TMP3, TMP1, idx);
LoadConstant(ARMEmitter::Size::i64Bit, TMP4, *(const uint16_t*)(OldCode + idx));
cmp(ARMEmitter::Size::i32Bit, TMP3, TMP4);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, TMP2, ARMEmitter::Condition::CC_EQ);
len -= 2;
idx += 2;
}
while (len >= 1)
{
ldrb(ARMEmitter::Reg::r2, ARMEmitter::Reg::r0, idx);
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r3, *(const uint8_t *)(OldCode + idx));
cmp(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r2, ARMEmitter::Reg::r3);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, ARMEmitter::Reg::r1, ARMEmitter::Condition::CC_EQ);
while (len >= 1) {
ldrb(TMP3, TMP1, idx);
LoadConstant(ARMEmitter::Size::i64Bit, TMP4, *(const uint8_t*)(OldCode + idx));
cmp(ARMEmitter::Size::i32Bit, TMP3, TMP4);
csel(ARMEmitter::Size::i64Bit, Dst, Dst, TMP2, ARMEmitter::Condition::CC_EQ);
len -= 1;
idx += 1;
}
}
DEF_OP(ThreadRemoveCodeEntry) {
PushDynamicRegsAndLR(TMP4);
SpillStaticRegs(TMP4);
// Arguments are passed as follows:
// X0: Thread
// X1: RIP
PushDynamicRegsAndLR(TMP1);
SpillStaticRegs(TMP1);
mov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, STATE.R());
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, Entry);
ldr(ARMEmitter::XReg::x2, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.ThreadRemoveCodeEntryFromJIT));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<void, void*, void*>(ARMEmitter::Reg::r2);
}
else {
} else {
blr(ARMEmitter::Reg::r2);
}
FillStaticRegs();
@@ -439,23 +394,34 @@ DEF_OP(ThreadRemoveCodeEntry) {
DEF_OP(CPUID) {
auto Op = IROp->C<IR::IROp_CPUID>();
PushDynamicRegsAndLR(TMP1);
SpillStaticRegs(TMP1);
mov(ARMEmitter::Size::i64Bit, TMP2, GetReg(Op->Function.ID()));
mov(ARMEmitter::Size::i64Bit, TMP3, GetReg(Op->Leaf.ID()));
PushDynamicRegsAndLR(TMP4);
SpillStaticRegs(TMP4);
// x0 = CPUID Handler
// x1 = CPUID Function
// x2 = CPUID Leaf
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.CPUIDObj));
ldr(ARMEmitter::XReg::x3, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.CPUIDFunction));
mov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, GetReg(Op->Function.ID()));
mov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r2, GetReg(Op->Leaf.ID()));
if (!TMP_ABIARGS) {
mov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, TMP2);
mov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r2, TMP3);
}
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<__uint128_t, void*, uint64_t, uint64_t>(ARMEmitter::Reg::r3);
}
else {
} else {
blr(ARMEmitter::Reg::r3);
}
if (!TMP_ABIARGS) {
mov(ARMEmitter::Size::i64Bit, TMP1, ARMEmitter::Reg::r0);
mov(ARMEmitter::Size::i64Bit, TMP2, ARMEmitter::Reg::r1);
}
FillStaticRegs();
PopDynamicRegsAndLR();
@@ -463,28 +429,32 @@ DEF_OP(CPUID) {
// Results are in x0, x1
// Results want to be in a i64v2 vector
auto Dst = GetRegPair(Node);
mov(ARMEmitter::Size::i64Bit, Dst.first, ARMEmitter::Reg::r0);
mov(ARMEmitter::Size::i64Bit, Dst.second, ARMEmitter::Reg::r1);
mov(ARMEmitter::Size::i64Bit, Dst.first, TMP1);
mov(ARMEmitter::Size::i64Bit, Dst.second, TMP2);
}
DEF_OP(XGETBV) {
DEF_OP(XGetBV) {
auto Op = IROp->C<IR::IROp_XGetBV>();
PushDynamicRegsAndLR(TMP1);
SpillStaticRegs(TMP1);
PushDynamicRegsAndLR(TMP4);
SpillStaticRegs(TMP4);
mov(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r1, GetReg(Op->Function.ID()));
// x0 = CPUID Handler
// x1 = XCR Function
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.CPUIDObj));
ldr(ARMEmitter::XReg::x2, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.XCRFunction));
mov(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r1, GetReg(Op->Function.ID()));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uint64_t, void*, uint32_t>(ARMEmitter::Reg::r2);
}
else {
} else {
blr(ARMEmitter::Reg::r2);
}
if (!TMP_ABIARGS) {
mov(ARMEmitter::Size::i64Bit, TMP1, ARMEmitter::Reg::r0);
}
FillStaticRegs();
PopDynamicRegsAndLR();
@@ -492,10 +462,9 @@ DEF_OP(XGETBV) {
// Results are in x0
// Results want to be in a i32v2 vector
auto Dst = GetRegPair(Node);
mov(ARMEmitter::Size::i32Bit, Dst.first, ARMEmitter::Reg::r0);
lsr(ARMEmitter::Size::i64Bit, Dst.second, ARMEmitter::Reg::r0, 32);
mov(ARMEmitter::Size::i32Bit, Dst.first, TMP1);
lsr(ARMEmitter::Size::i64Bit, Dst.second, TMP1, 32);
}
#undef DEF_OP
}
} // namespace FEXCore::CPU
@@ -5,11 +5,10 @@ tags: backend|arm64
$end_info$
*/
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(VInsGPR) {
const auto Op = IROp->C<IR::IROp_VInsGPR>();
const auto OpSize = IROp->Size;
@@ -18,11 +17,7 @@ DEF_OP(VInsGPR) {
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
LOGMAN_THROW_AA_FMT(ElementSize == 8 || ElementSize == 4 || ElementSize == 2 || ElementSize == 1, "Unexpected {} size", __func__);
const auto SubEmitSize = ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
ElementSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
const auto SubEmitSize = ConvertSubRegSize8(IROp);
const auto ElementsPer128Bit = 16 / ElementSize;
const auto Dst = GetVReg(Node);
@@ -65,7 +60,7 @@ DEF_OP(VInsGPR) {
// Inserts the GPR value into the given V register.
// Also automatically adjusts the index in the case of using the
// moved upper lane.
const auto Insert = [&](const FEXCore::ARMEmitter::VRegister& reg, int index) {
const auto Insert = [&](const ARMEmitter::VRegister& reg, int index) {
if (InUpperLane) {
index -= ElementsPer128Bit;
}
@@ -94,21 +89,17 @@ DEF_OP(VCastFromGPR) {
auto Src = GetReg(Op->Src.ID());
switch (Op->Header.ElementSize) {
case 1:
uxtb(ARMEmitter::Size::i32Bit, TMP1, Src);
fmov(ARMEmitter::Size::i32Bit, Dst.S(), TMP1);
break;
case 2:
uxth(ARMEmitter::Size::i32Bit, TMP1, Src);
fmov(ARMEmitter::Size::i32Bit, Dst.S(), TMP1);
break;
case 4:
fmov(ARMEmitter::Size::i32Bit, Dst.S(), Src);
break;
case 8:
fmov(ARMEmitter::Size::i64Bit, Dst.D(), Src);
break;
default: LOGMAN_MSG_A_FMT("Unknown castGPR element size: {}", Op->Header.ElementSize);
case 1:
uxtb(ARMEmitter::Size::i32Bit, TMP1, Src);
fmov(ARMEmitter::Size::i32Bit, Dst.S(), TMP1);
break;
case 2:
uxth(ARMEmitter::Size::i32Bit, TMP1, Src);
fmov(ARMEmitter::Size::i32Bit, Dst.S(), TMP1);
break;
case 4: fmov(ARMEmitter::Size::i32Bit, Dst.S(), Src); break;
case 8: fmov(ARMEmitter::Size::i64Bit, Dst.D(), Src); break;
default: LOGMAN_MSG_A_FMT("Unknown castGPR element size: {}", Op->Header.ElementSize);
}
}
@@ -120,16 +111,7 @@ DEF_OP(VDupFromGPR) {
const auto Src = GetReg(Op->Src.ID());
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto ElementSize = IROp->ElementSize;
LOGMAN_THROW_AA_FMT(ElementSize == 8 || ElementSize == 4 || ElementSize == 2 || ElementSize == 1,
"Unexpected {} element size: {}", __func__, ElementSize);
const auto SubEmitSize =
ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
ElementSize == 1 ? ARMEmitter::SubRegSize::i8Bit : ARMEmitter::SubRegSize::i8Bit;
const auto SubEmitSize = ConvertSubRegSize8(IROp);
if (HostSupportsSVE256 && Is256Bit) {
dup(SubEmitSize, Dst.Z(), Src);
@@ -148,34 +130,33 @@ DEF_OP(Float_FromGPR_S) {
auto Src = GetReg(Op->Src.ID());
switch (Conv) {
case 0x0204: { // Half <- int32_t
scvtf(ARMEmitter::Size::i32Bit, Dst.H(), Src);
break;
}
case 0x0208: { // Half <- int64_t
scvtf(ARMEmitter::Size::i64Bit, Dst.H(), Src);
break;
}
case 0x0404: { // Float <- int32_t
scvtf(ARMEmitter::Size::i32Bit, Dst.S(), Src);
break;
}
case 0x0408: { // Float <- int64_t
scvtf(ARMEmitter::Size::i64Bit, Dst.S(), Src);
break;
}
case 0x0804: { // Double <- int32_t
scvtf(ARMEmitter::Size::i32Bit, Dst.D(), Src);
break;
}
case 0x0808: { // Double <- int64_t
scvtf(ARMEmitter::Size::i64Bit, Dst.D(), Src);
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled conversion mask: Mask=0x{:04x}, ElementSize={}, SrcElementSize={}",
Conv, ElementSize, Op->SrcElementSize);
break;
case 0x0204: { // Half <- int32_t
scvtf(ARMEmitter::Size::i32Bit, Dst.H(), Src);
break;
}
case 0x0208: { // Half <- int64_t
scvtf(ARMEmitter::Size::i64Bit, Dst.H(), Src);
break;
}
case 0x0404: { // Float <- int32_t
scvtf(ARMEmitter::Size::i32Bit, Dst.S(), Src);
break;
}
case 0x0408: { // Float <- int64_t
scvtf(ARMEmitter::Size::i64Bit, Dst.S(), Src);
break;
}
case 0x0804: { // Double <- int32_t
scvtf(ARMEmitter::Size::i32Bit, Dst.D(), Src);
break;
}
case 0x0808: { // Double <- int64_t
scvtf(ARMEmitter::Size::i64Bit, Dst.D(), Src);
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled conversion mask: Mask=0x{:04x}, ElementSize={}, SrcElementSize={}", Conv, ElementSize, Op->SrcElementSize);
break;
}
}
@@ -187,31 +168,31 @@ DEF_OP(Float_FToF) {
auto Src = GetVReg(Op->Scalar.ID());
switch (Conv) {
case 0x0204: { // Half <- Float
fcvt(Dst.H(), Src.S());
break;
}
case 0x0208: { // Half <- Double
fcvt(Dst.H(), Src.D());
break;
}
case 0x0402: { // Float <- Half
fcvt(Dst.S(), Src.H());
break;
}
case 0x0802: { // Double <- Half
fcvt(Dst.D(), Src.H());
break;
}
case 0x0804: { // Double <- Float
fcvt(Dst.D(), Src.S());
break;
}
case 0x0408: { // Float <- Double
fcvt(Dst.S(), Src.D());
break;
}
default: LOGMAN_MSG_A_FMT("Unknown FCVT sizes: 0x{:x}", Conv);
case 0x0204: { // Half <- Float
fcvt(Dst.H(), Src.S());
break;
}
case 0x0208: { // Half <- Double
fcvt(Dst.H(), Src.D());
break;
}
case 0x0402: { // Float <- Half
fcvt(Dst.S(), Src.H());
break;
}
case 0x0802: { // Double <- Half
fcvt(Dst.D(), Src.H());
break;
}
case 0x0804: { // Double <- Float
fcvt(Dst.D(), Src.S());
break;
}
case 0x0408: { // Float <- Double
fcvt(Dst.S(), Src.D());
break;
}
default: LOGMAN_MSG_A_FMT("Unknown FCVT sizes: 0x{:x}", Conv);
}
}
@@ -220,13 +201,9 @@ DEF_OP(Vector_SToF) {
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
LOGMAN_THROW_AA_FMT(ElementSize == 8 || ElementSize == 4 || ElementSize == 2, "Unexpected {} size", __func__);
const auto SubEmitSize = ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit : ARMEmitter::SubRegSize::i16Bit;
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
if (HostSupportsSVE256 && Is256Bit) {
@@ -236,19 +213,15 @@ DEF_OP(Vector_SToF) {
if (OpSize == ElementSize) {
if (ElementSize == 8) {
scvtf(ARMEmitter::ScalarRegSize::i64Bit, Dst.D(), Vector.D());
}
else if (ElementSize == 4) {
} else if (ElementSize == 4) {
scvtf(ARMEmitter::ScalarRegSize::i32Bit, Dst.S(), Vector.S());
}
else {
} else {
scvtf(ARMEmitter::ScalarRegSize::i16Bit, Dst.H(), Vector.H());
}
}
else {
} else {
if (OpSize == 8) {
scvtf(SubEmitSize, Dst.D(), Vector.D());
}
else {
} else {
scvtf(SubEmitSize, Dst.Q(), Vector.Q());
}
}
@@ -260,13 +233,9 @@ DEF_OP(Vector_FToZS) {
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
LOGMAN_THROW_AA_FMT(ElementSize == 8 || ElementSize == 4 || ElementSize == 2, "Unexpected {} size", __func__);
const auto SubEmitSize = ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit : ARMEmitter::SubRegSize::i16Bit;
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
if (HostSupportsSVE256 && Is256Bit) {
@@ -276,19 +245,15 @@ DEF_OP(Vector_FToZS) {
if (OpSize == ElementSize) {
if (ElementSize == 8) {
fcvtzs(ARMEmitter::ScalarRegSize::i64Bit, Dst.D(), Vector.D());
}
else if (ElementSize == 4) {
} else if (ElementSize == 4) {
fcvtzs(ARMEmitter::ScalarRegSize::i32Bit, Dst.S(), Vector.S());
}
else {
} else {
fcvtzs(ARMEmitter::ScalarRegSize::i16Bit, Dst.H(), Vector.H());
}
}
else {
} else {
if (OpSize == 8) {
fcvtzs(SubEmitSize, Dst.D(), Vector.D());
}
else {
} else {
fcvtzs(SubEmitSize, Dst.Q(), Vector.Q());
}
}
@@ -299,13 +264,8 @@ DEF_OP(Vector_FToS) {
const auto Op = IROp->C<IR::IROp_Vector_FToS>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
LOGMAN_THROW_AA_FMT(ElementSize == 8 || ElementSize == 4 || ElementSize == 2, "Unexpected {} size", __func__);
const auto SubEmitSize = ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit : ARMEmitter::SubRegSize::i16Bit;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
@@ -320,8 +280,7 @@ DEF_OP(Vector_FToS) {
if (OpSize == 8) {
frinti(SubEmitSize, Dst.D(), Vector.D());
fcvtzs(SubEmitSize, Dst.D(), Dst.D());
}
else {
} else {
frinti(SubEmitSize, Dst.Q(), Vector.Q());
fcvtzs(SubEmitSize, Dst.Q(), Dst.Q());
}
@@ -333,14 +292,10 @@ DEF_OP(Vector_FToF) {
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Conv = (ElementSize << 8) | Op->SrcElementSize;
LOGMAN_THROW_AA_FMT(ElementSize == 8 || ElementSize == 4 || ElementSize == 2, "Unexpected {} size", __func__);
const auto SubEmitSize = ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit : ARMEmitter::SubRegSize::i16Bit;
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
@@ -361,46 +316,75 @@ DEF_OP(Vector_FToF) {
const auto Mask = PRED_TMP_32B.Merging();
switch (Conv) {
case 0x0402: { // Float <- Half
zip1(FEXCore::ARMEmitter::SubRegSize::i16Bit, Dst.Z(), Vector.Z(), Vector.Z());
fcvtlt(FEXCore::ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Mask, Dst.Z());
break;
}
case 0x0804: { // Double <- Float
zip1(FEXCore::ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Vector.Z(), Vector.Z());
fcvtlt(FEXCore::ARMEmitter::SubRegSize::i64Bit, Dst.Z(), Mask, Dst.Z());
break;
}
case 0x0204: { // Half <- Float
fcvtnt(FEXCore::ARMEmitter::SubRegSize::i16Bit, Dst.Z(), Mask, Vector.Z());
uzp2(FEXCore::ARMEmitter::SubRegSize::i16Bit, Dst.Z(), Dst.Z(), Dst.Z());
break;
}
case 0x0408: { // Float <- Double
fcvtnt(FEXCore::ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Mask, Vector.Z());
uzp2(FEXCore::ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Dst.Z(), Dst.Z());
break;
}
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToF Type : 0x{:04x}", Conv);
break;
case 0x0402: { // Float <- Half
zip1(ARMEmitter::SubRegSize::i16Bit, Dst.Z(), Vector.Z(), Vector.Z());
fcvtlt(ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Mask, Dst.Z());
break;
}
case 0x0804: { // Double <- Float
zip1(ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Vector.Z(), Vector.Z());
fcvtlt(ARMEmitter::SubRegSize::i64Bit, Dst.Z(), Mask, Dst.Z());
break;
}
case 0x0204: { // Half <- Float
fcvtnt(ARMEmitter::SubRegSize::i16Bit, Dst.Z(), Mask, Vector.Z());
uzp2(ARMEmitter::SubRegSize::i16Bit, Dst.Z(), Dst.Z(), Dst.Z());
break;
}
case 0x0408: { // Float <- Double
fcvtnt(ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Mask, Vector.Z());
uzp2(ARMEmitter::SubRegSize::i32Bit, Dst.Z(), Dst.Z(), Dst.Z());
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToF Type : 0x{:04x}", Conv); break;
}
} else {
switch (Conv) {
case 0x0402: // Float <- Half
case 0x0804: { // Double <- Float
fcvtl(SubEmitSize, Dst.D(), Vector.D());
break;
}
case 0x0204: // Half <- Float
case 0x0408: { // Float <- Double
fcvtn(SubEmitSize, Dst.D(), Vector.D());
break;
}
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToF Type : 0x{:04x}", Conv);
break;
case 0x0402: // Float <- Half
case 0x0804: { // Double <- Float
fcvtl(SubEmitSize, Dst.D(), Vector.D());
break;
}
case 0x0204: // Half <- Float
case 0x0408: { // Float <- Double
fcvtn(SubEmitSize, Dst.D(), Vector.D());
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToF Type : 0x{:04x}", Conv); break;
}
}
}
DEF_OP(VFCVTL2) {
const auto Op = IROp->C<IR::IROp_VFCVTL2>();
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
fcvtl2(SubEmitSize, Dst.D(), Vector.D());
}
DEF_OP(VFCVTN2) {
const auto Op = IROp->C<IR::IROp_VFCVTN2>();
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Dst = GetVReg(Node);
const auto VectorLower = GetVReg(Op->VectorLower.ID());
const auto VectorUpper = GetVReg(Op->VectorUpper.ID());
auto Lower = VectorLower;
if (Dst != VectorLower) {
mov(VTMP1.Q(), VectorLower.Q());
Lower = VTMP1;
}
fcvtn2(SubEmitSize, Lower.Q(), VectorUpper.Q());
if (Dst != VectorLower) {
mov(Dst.Q(), Lower.Q());
}
}
@@ -409,12 +393,8 @@ DEF_OP(Vector_FToI) {
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
LOGMAN_THROW_AA_FMT(ElementSize == 8 || ElementSize == 4 || ElementSize == 2, "Unexpected {} size", __func__);
const auto SubEmitSize = ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit : ARMEmitter::SubRegSize::i16Bit;
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
@@ -423,82 +403,51 @@ DEF_OP(Vector_FToI) {
const auto Mask = PRED_TMP_32B.Merging();
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
frintn(SubEmitSize, Dst.Z(), Mask, Vector.Z());
break;
case FEXCore::IR::Round_Negative_Infinity.Val:
frintm(SubEmitSize, Dst.Z(), Mask, Vector.Z());
break;
case FEXCore::IR::Round_Positive_Infinity.Val:
frintp(SubEmitSize, Dst.Z(), Mask, Vector.Z());
break;
case FEXCore::IR::Round_Towards_Zero.Val:
frintz(SubEmitSize, Dst.Z(), Mask, Vector.Z());
break;
case FEXCore::IR::Round_Host.Val:
frinti(SubEmitSize, Dst.Z(), Mask, Vector.Z());
break;
case FEXCore::IR::Round_Nearest.Val: frintn(SubEmitSize, Dst.Z(), Mask, Vector.Z()); break;
case FEXCore::IR::Round_Negative_Infinity.Val: frintm(SubEmitSize, Dst.Z(), Mask, Vector.Z()); break;
case FEXCore::IR::Round_Positive_Infinity.Val: frintp(SubEmitSize, Dst.Z(), Mask, Vector.Z()); break;
case FEXCore::IR::Round_Towards_Zero.Val: frintz(SubEmitSize, Dst.Z(), Mask, Vector.Z()); break;
case FEXCore::IR::Round_Host.Val: frinti(SubEmitSize, Dst.Z(), Mask, Vector.Z()); break;
}
} else {
const auto IsScalar = ElementSize == OpSize;
if (IsScalar) {
// Since we have multiple overloads of the same name (e.g.
// frinti having AdvSIMD, AdvSIMD scalar, and an SVE version),
// we can't just use a lambda without some seriously ugly casting.
// This is fairly self-contained otherwise.
#define ROUNDING_FN(name) \
if (ElementSize == 2) { \
name(Dst.H(), Vector.H()); \
} else if (ElementSize == 4) { \
name(Dst.S(), Vector.S()); \
} else if (ElementSize == 8) { \
name(Dst.D(), Vector.D()); \
} else { \
FEX_UNREACHABLE; \
}
// Since we have multiple overloads of the same name (e.g.
// frinti having AdvSIMD, AdvSIMD scalar, and an SVE version),
// we can't just use a lambda without some seriously ugly casting.
// This is fairly self-contained otherwise.
#define ROUNDING_FN(name) \
if (ElementSize == 2) { \
name(Dst.H(), Vector.H()); \
} else if (ElementSize == 4) { \
name(Dst.S(), Vector.S()); \
} else if (ElementSize == 8) { \
name(Dst.D(), Vector.D()); \
} else { \
FEX_UNREACHABLE; \
}
switch (Op->Round) {
case IR::Round_Nearest.Val:
ROUNDING_FN(frintn);
break;
case IR::Round_Negative_Infinity.Val:
ROUNDING_FN(frintm);
break;
case IR::Round_Positive_Infinity.Val:
ROUNDING_FN(frintp);
break;
case IR::Round_Towards_Zero.Val:
ROUNDING_FN(frintz);
break;
case IR::Round_Host.Val:
ROUNDING_FN(frinti);
break;
case IR::Round_Nearest.Val: ROUNDING_FN(frintn); break;
case IR::Round_Negative_Infinity.Val: ROUNDING_FN(frintm); break;
case IR::Round_Positive_Infinity.Val: ROUNDING_FN(frintp); break;
case IR::Round_Towards_Zero.Val: ROUNDING_FN(frintz); break;
case IR::Round_Host.Val: ROUNDING_FN(frinti); break;
}
#undef ROUNDING_FN
#undef ROUNDING_FN
} else {
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
frintn(SubEmitSize, Dst.Q(), Vector.Q());
break;
case FEXCore::IR::Round_Negative_Infinity.Val:
frintm(SubEmitSize, Dst.Q(), Vector.Q());
break;
case FEXCore::IR::Round_Positive_Infinity.Val:
frintp(SubEmitSize, Dst.Q(), Vector.Q());
break;
case FEXCore::IR::Round_Towards_Zero.Val:
frintz(SubEmitSize, Dst.Q(), Vector.Q());
break;
case FEXCore::IR::Round_Host.Val:
frinti(SubEmitSize, Dst.Q(), Vector.Q());
break;
case FEXCore::IR::Round_Nearest.Val: frintn(SubEmitSize, Dst.Q(), Vector.Q()); break;
case FEXCore::IR::Round_Negative_Infinity.Val: frintm(SubEmitSize, Dst.Q(), Vector.Q()); break;
case FEXCore::IR::Round_Positive_Infinity.Val: frintp(SubEmitSize, Dst.Q(), Vector.Q()); break;
case FEXCore::IR::Round_Towards_Zero.Val: frintz(SubEmitSize, Dst.Q(), Vector.Q()); break;
case FEXCore::IR::Round_Host.Val: frinti(SubEmitSize, Dst.Q(), Vector.Q()); break;
}
}
}
}
#undef DEF_OP
}
} // namespace FEXCore::CPU
@@ -5,19 +5,17 @@ tags: backend|arm64
$end_info$
*/
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/IR/Passes/RegisterAllocationPass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(AESImc) {
DEF_OP(VAESImc) {
auto Op = IROp->C<IR::IROp_VAESImc>();
aesimc(GetVReg(Node), GetVReg(Op->Vector.ID()));
}
DEF_OP(AESEnc) {
DEF_OP(VAESEnc) {
const auto Op = IROp->C<IR::IROp_VAESEnc>();
const auto OpSize = IROp->Size;
@@ -26,8 +24,7 @@ DEF_OP(AESEnc) {
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE,
"Currently only supports 128-bit operations.");
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
@@ -35,8 +32,7 @@ DEF_OP(AESEnc) {
aese(Dst.Q(), ZeroReg.Q());
aesmc(Dst.Q(), Dst.Q());
eor(Dst.Q(), Dst.Q(), Key.Q());
}
else {
} else {
mov(VTMP1.Q(), State.Q());
aese(VTMP1, ZeroReg.Q());
aesmc(VTMP1, VTMP1);
@@ -44,7 +40,7 @@ DEF_OP(AESEnc) {
}
}
DEF_OP(AESEncLast) {
DEF_OP(VAESEncLast) {
const auto Op = IROp->C<IR::IROp_VAESEncLast>();
const auto OpSize = IROp->Size;
@@ -53,23 +49,21 @@ DEF_OP(AESEncLast) {
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE,
"Currently only supports 128-bit operations.");
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
// This matches the common case of XMM AES.
aese(Dst.Q(), ZeroReg.Q());
eor(Dst.Q(), Dst.Q(), Key.Q());
}
else {
} else {
mov(VTMP1.Q(), State.Q());
aese(VTMP1, ZeroReg.Q());
eor(Dst.Q(), VTMP1.Q(), Key.Q());
}
}
DEF_OP(AESDec) {
DEF_OP(VAESDec) {
const auto Op = IROp->C<IR::IROp_VAESDec>();
const auto OpSize = IROp->Size;
@@ -78,8 +72,7 @@ DEF_OP(AESDec) {
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE,
"Currently only supports 128-bit operations.");
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
@@ -87,8 +80,7 @@ DEF_OP(AESDec) {
aesd(Dst.Q(), ZeroReg.Q());
aesimc(Dst.Q(), Dst.Q());
eor(Dst.Q(), Dst.Q(), Key.Q());
}
else {
} else {
mov(VTMP1.Q(), State.Q());
aesd(VTMP1, ZeroReg.Q());
aesimc(VTMP1, VTMP1);
@@ -96,7 +88,7 @@ DEF_OP(AESDec) {
}
}
DEF_OP(AESDecLast) {
DEF_OP(VAESDecLast) {
const auto Op = IROp->C<IR::IROp_VAESDecLast>();
const auto OpSize = IROp->Size;
@@ -105,23 +97,21 @@ DEF_OP(AESDecLast) {
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE,
"Currently only supports 128-bit operations.");
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
// This matches the common case of XMM AES.
aesd(Dst.Q(), ZeroReg.Q());
eor(Dst.Q(), Dst.Q(), Key.Q());
}
else {
} else {
mov(VTMP1.Q(), State.Q());
aesd(VTMP1, ZeroReg.Q());
eor(Dst.Q(), VTMP1.Q(), Key.Q());
}
}
DEF_OP(AESKeyGenAssist) {
DEF_OP(VAESKeyGenAssist) {
auto Op = IROp->C<IR::IROp_VAESKeyGenAssist>();
const auto Dst = GetVReg(Node);
const auto Src = GetVReg(Op->Src.ID());
@@ -149,8 +139,7 @@ DEF_OP(AESKeyGenAssist) {
LoadConstant(ARMEmitter::Size::i64Bit, TMP1, static_cast<uint64_t>(Op->RCON) << 32);
dup(ARMEmitter::SubRegSize::i64Bit, VTMP2.Q(), TMP1);
eor(Dst.Q(), Dst.Q(), VTMP2.Q());
}
else {
} else {
tbl(Dst.Q(), Dst.Q(), Swizzle.Q());
}
}
@@ -163,19 +152,36 @@ DEF_OP(CRC32) {
const auto Src2 = GetReg(Op->Src2.ID());
switch (Op->SrcSize) {
case 1:
crc32cb(Dst.W(), Src1.W(), Src2.W());
break;
case 2:
crc32ch(Dst.W(), Src1.W(), Src2.W());
break;
case 4:
crc32cw(Dst.W(), Src1.W(), Src2.W());
break;
case 8:
crc32cx(Dst.X(), Src1.X(), Src2.X());
break;
default: LOGMAN_MSG_A_FMT("Unknown CRC32 size: {}", Op->SrcSize);
case 1: crc32cb(Dst.W(), Src1.W(), Src2.W()); break;
case 2: crc32ch(Dst.W(), Src1.W(), Src2.W()); break;
case 4: crc32cw(Dst.W(), Src1.W(), Src2.W()); break;
case 8: crc32cx(Dst.X(), Src1.X(), Src2.X()); break;
default: LOGMAN_MSG_A_FMT("Unknown CRC32 size: {}", Op->SrcSize);
}
}
DEF_OP(VSha1H) {
auto Op = IROp->C<IR::IROp_VSha1H>();
const auto Dst = GetVReg(Node);
const auto Src = GetVReg(Op->Src.ID());
sha1h(Dst.S(), Src.S());
}
DEF_OP(VSha256U0) {
auto Op = IROp->C<IR::IROp_VSha256U0>();
const auto Dst = GetVReg(Node);
const auto Src1 = GetVReg(Op->Src1.ID());
const auto Src2 = GetVReg(Op->Src2.ID());
if (Dst == Src1) {
sha256su0(Dst, Src2);
} else {
mov(VTMP1.Q(), Src1.Q());
sha256su0(VTMP1, Src2);
mov(Dst.Q(), VTMP1.Q());
}
}
@@ -183,17 +189,14 @@ DEF_OP(PCLMUL) {
const auto Op = IROp->C<IR::IROp_PCLMUL>();
const auto OpSize = IROp->Size;
const auto Dst = GetVReg(Node);
const auto Dst = GetVReg(Node);
const auto Src1 = GetVReg(Op->Src1.ID());
const auto Src2 = GetVReg(Op->Src2.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE,
"Currently only supports 128-bit operations.");
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
switch (Op->Selector) {
case 0b00000000:
pmull(ARMEmitter::SubRegSize::i128Bit, Dst.D(), Src1.D(), Src2.D());
break;
case 0b00000000: pmull(ARMEmitter::SubRegSize::i128Bit, Dst.D(), Src1.D(), Src2.D()); break;
case 0b00000001:
dup(ARMEmitter::SubRegSize::i64Bit, VTMP1.Q(), Src1.Q(), 1);
pmull(ARMEmitter::SubRegSize::i128Bit, Dst.D(), VTMP1.D(), Src2.D());
@@ -202,14 +205,10 @@ DEF_OP(PCLMUL) {
dup(ARMEmitter::SubRegSize::i64Bit, VTMP1.Q(), Src2.Q(), 1);
pmull(ARMEmitter::SubRegSize::i128Bit, Dst.D(), VTMP1.D(), Src1.D());
break;
case 0b00010001:
pmull2(ARMEmitter::SubRegSize::i128Bit, Dst.Q(), Src1.Q(), Src2.Q());
break;
default:
LOGMAN_MSG_A_FMT("Unknown PCLMUL selector: {}", Op->Selector);
break;
case 0b00010001: pmull2(ARMEmitter::SubRegSize::i128Bit, Dst.Q(), Src1.Q(), Src2.Q()); break;
default: LOGMAN_MSG_A_FMT("Unknown PCLMUL selector: {}", Op->Selector); break;
}
}
#undef DEF_OP
}
} // namespace FEXCore::CPU
@@ -8,12 +8,11 @@ $end_info$
#include "Interface/Core/JIT/Arm64/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(GetHostFlag) {
auto Op = IROp->C<IR::IROp_GetHostFlag>();
ubfx(ARMEmitter::Size::i64Bit, GetReg(Node), GetReg(Op->Value.ID()), Op->Flag, 1);
}
#undef DEF_OP
}
} // namespace FEXCore::CPU
File diff suppressed because it is too large. Load diff
+241 -360
View File
@@ -8,66 +8,84 @@ $end_info$
#pragma once
#include "Interface/Core/ArchHelpers/Arm64Emitter.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/CPUBackend.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include "Interface/IR/IR.h"
#include "Interface/IR/IntrusiveIRList.h"
#include "Interface/IR/RegisterAllocationData.h"
#include <aarch64/assembler-aarch64.h>
#include <aarch64/disasm-aarch64.h>
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Core/CPUBackend.h>
#include <FEXCore/IR/IR.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/IR/RegisterAllocationData.h>
#include <FEXCore/fextl/map.h>
#include <FEXCore/fextl/string.h>
#include <FEXCore/fextl/vector.h>
#include <CodeEmitter/Emitter.h>
#include <array>
#include <cstdint>
#include <utility>
#include <variant>
namespace FEXCore::Core {
struct InternalThreadState;
struct InternalThreadState;
}
namespace FEXCore::CPU {
class Arm64JITCore final : public CPUBackend, public Arm64Emitter {
class Arm64JITCore final : public CPUBackend, public Arm64Emitter {
public:
explicit Arm64JITCore(FEXCore::Context::ContextImpl *ctx,
FEXCore::Core::InternalThreadState *Thread);
explicit Arm64JITCore(FEXCore::Context::ContextImpl* ctx, FEXCore::Core::InternalThreadState* Thread);
~Arm64JITCore() override;
[[nodiscard]] fextl::string GetName() override { return "JIT"; }
[[nodiscard]]
fextl::string GetName() override {
return "JIT";
}
[[nodiscard]] CPUBackend::CompiledCode CompileCode(uint64_t Entry,
FEXCore::IR::IRListView const *IR,
FEXCore::Core::DebugData *DebugData,
FEXCore::IR::RegisterAllocationData *RAData, bool GDBEnabled) override;
[[nodiscard]]
CPUBackend::CompiledCode CompileCode(uint64_t Entry, const FEXCore::IR::IRListView* IR, FEXCore::Core::DebugData* DebugData,
const FEXCore::IR::RegisterAllocationData* RAData) override;
[[nodiscard]] void *MapRegion(void* HostPtr, uint64_t, uint64_t) override { return HostPtr; }
[[nodiscard]]
void* MapRegion(void* HostPtr, uint64_t, uint64_t) override {
return HostPtr;
}
[[nodiscard]] bool NeedsOpDispatch() override { return true; }
[[nodiscard]]
bool NeedsOpDispatch() override {
return true;
}
void ClearCache() override;
void ClearRelocations() override { Relocations.clear(); }
void ClearRelocations() override {
Relocations.clear();
}
private:
FEX_CONFIG_OPT(ParanoidTSO, PARANOIDTSO);
FEX_CONFIG_OPT(VectorTSOEnabled, VECTORTSOENABLED);
FEX_CONFIG_OPT(MemcpySetTSOEnabled, MEMCPYSETTSOENABLED);
const bool HostSupportsSVE128{};
const bool HostSupportsSVE256{};
const bool HostSupportsSVE128 {};
const bool HostSupportsSVE256 {};
const bool HostSupportsAVX256 {};
const bool HostSupportsRPRES {};
const bool HostSupportsAFP {};
ARMEmitter::BiDirectionalLabel *PendingTargetLabel;
FEXCore::Context::ContextImpl *CTX;
FEXCore::IR::IRListView const *IR;
ARMEmitter::BiDirectionalLabel* PendingTargetLabel;
FEXCore::Context::ContextImpl* CTX;
const FEXCore::IR::IRListView* IR;
uint64_t Entry;
CPUBackend::CompiledCode CodeData{};
CPUBackend::CompiledCode CodeData {};
fextl::map<IR::NodeID, ARMEmitter::BiDirectionalLabel> JumpTargets;
[[nodiscard]] FEXCore::ARMEmitter::Register GetReg(IR::NodeID Node) const {
[[nodiscard]]
ARMEmitter::Register GetReg(IR::NodeID Node) const {
const auto Reg = GetPhys(Node);
LOGMAN_THROW_AA_FMT(Reg.Class == IR::GPRFixedClass.Val || Reg.Class == IR::GPRClass.Val, "Unexpected Class: {}", Reg.Class);
@@ -81,7 +99,8 @@ private:
FEX_UNREACHABLE;
}
[[nodiscard]] FEXCore::ARMEmitter::VRegister GetVReg(IR::NodeID Node) const {
[[nodiscard]]
ARMEmitter::VRegister GetVReg(IR::NodeID Node) const {
const auto Reg = GetPhys(Node);
LOGMAN_THROW_AA_FMT(Reg.Class == IR::FPRFixedClass.Val || Reg.Class == IR::FPRClass.Val, "Unexpected Class: {}", Reg.Class);
@@ -95,17 +114,20 @@ private:
FEX_UNREACHABLE;
}
[[nodiscard]] std::pair<FEXCore::ARMEmitter::Register, FEXCore::ARMEmitter::Register> GetRegPair(IR::NodeID Node) const {
[[nodiscard]]
std::pair<ARMEmitter::Register, ARMEmitter::Register> GetRegPair(IR::NodeID Node) const {
const auto Reg = GetPhys(Node);
LOGMAN_THROW_AA_FMT(Reg.Class == IR::GPRPairClass.Val, "Unexpected Class: {}", Reg.Class);
return GeneralPairRegisters[Reg.Reg];
return std::make_pair(GeneralRegisters[Reg.Reg], GeneralRegisters[Reg.Reg + 1]);
}
[[nodiscard]] FEXCore::IR::RegisterClassType GetRegClass(IR::NodeID Node) const;
[[nodiscard]]
FEXCore::IR::RegisterClassType GetRegClass(IR::NodeID Node) const;
[[nodiscard]] IR::PhysicalRegister GetPhys(IR::NodeID Node) const {
[[nodiscard]]
IR::PhysicalRegister GetPhys(IR::NodeID Node) const {
auto PhyReg = RAData->GetNodeRegister(Node);
LOGMAN_THROW_A_FMT(!PhyReg.IsInvalid(), "Couldn't Allocate register for node: ssa{}. Class: {}", Node, PhyReg.Class);
@@ -113,29 +135,144 @@ private:
return PhyReg;
}
[[nodiscard]] bool IsFPR(IR::NodeID Node) const;
[[nodiscard]] bool IsGPR(IR::NodeID Node) const;
[[nodiscard]] bool IsGPRPair(IR::NodeID Node) const;
[[nodiscard]]
ARMEmitter::Register GetZeroableReg(IR::OrderedNodeWrapper Src) const {
uint64_t Const;
if (IsInlineConstant(Src, &Const)) {
LOGMAN_THROW_AA_FMT(Const == 0, "Only valid constant");
return ARMEmitter::Reg::zr;
} else {
return GetReg(Src.ID());
}
}
[[nodiscard]] FEXCore::ARMEmitter::ExtendedMemOperand GenerateMemOperand(uint8_t AccessSize,
FEXCore::ARMEmitter::Register Base,
IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType,
uint8_t OffsetScale);
// Converts IR-base shift type to ARMEmitter shift type.
// Will be a no-op, only a type conversion since the two definitions match.
[[nodiscard]]
ARMEmitter::ShiftType ConvertIRShiftType(IR::ShiftType Shift) const {
return Shift == IR::ShiftType::LSL ? ARMEmitter::ShiftType::LSL :
Shift == IR::ShiftType::LSR ? ARMEmitter::ShiftType::LSR :
Shift == IR::ShiftType::ASR ? ARMEmitter::ShiftType::ASR :
ARMEmitter::ShiftType::ROR;
}
[[nodiscard]]
ARMEmitter::Size ConvertSize(const IR::IROp_Header* Op) {
return Op->Size == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
}
[[nodiscard]]
ARMEmitter::Size ConvertSize48(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->Size == 4 || Op->Size == 8, "Invalid size");
return ConvertSize(Op);
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize16(uint8_t ElementSize) {
LOGMAN_THROW_AA_FMT(ElementSize == 1 || ElementSize == 2 || ElementSize == 4 || ElementSize == 8 || ElementSize == 16, "Invalid size");
return ElementSize == 1 ? ARMEmitter::SubRegSize::i8Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ARMEmitter::SubRegSize::i128Bit;
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize16(const IR::IROp_Header* Op) {
return ConvertSubRegSize16(Op->ElementSize);
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize8(uint8_t ElementSize) {
LOGMAN_THROW_AA_FMT(ElementSize != 16, "Invalid size");
return ConvertSubRegSize16(ElementSize);
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize8(const IR::IROp_Header* Op) {
return ConvertSubRegSize8(Op->ElementSize);
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize4(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 8, "Invalid size");
return ConvertSubRegSize8(Op);
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize248(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 1, "Invalid size");
return ConvertSubRegSize8(Op);
}
[[nodiscard]]
ARMEmitter::VectorRegSizePair ConvertSubRegSizePair16(const IR::IROp_Header* Op) {
return ARMEmitter::ToVectorSizePair(ConvertSubRegSize16(Op));
}
[[nodiscard]]
ARMEmitter::VectorRegSizePair ConvertSubRegSizePair8(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 16, "Invalid size");
return ConvertSubRegSizePair16(Op);
}
[[nodiscard]]
ARMEmitter::VectorRegSizePair ConvertSubRegSizePair248(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 1, "Invalid size");
return ConvertSubRegSizePair8(Op);
}
[[nodiscard]]
ARMEmitter::Condition MapCC(IR::CondClassType Cond) {
switch (Cond.Val) {
case FEXCore::IR::COND_EQ: return ARMEmitter::Condition::CC_EQ;
case FEXCore::IR::COND_NEQ: return ARMEmitter::Condition::CC_NE;
case FEXCore::IR::COND_SGE: return ARMEmitter::Condition::CC_GE;
case FEXCore::IR::COND_SLT: return ARMEmitter::Condition::CC_LT;
case FEXCore::IR::COND_SGT: return ARMEmitter::Condition::CC_GT;
case FEXCore::IR::COND_SLE: return ARMEmitter::Condition::CC_LE;
case FEXCore::IR::COND_UGE: return ARMEmitter::Condition::CC_CS;
case FEXCore::IR::COND_ULT: return ARMEmitter::Condition::CC_CC;
case FEXCore::IR::COND_UGT: return ARMEmitter::Condition::CC_HI;
case FEXCore::IR::COND_ULE: return ARMEmitter::Condition::CC_LS;
case FEXCore::IR::COND_FLU: return ARMEmitter::Condition::CC_LT;
case FEXCore::IR::COND_FGE: return ARMEmitter::Condition::CC_GE;
case FEXCore::IR::COND_FLEU: return ARMEmitter::Condition::CC_LE;
case FEXCore::IR::COND_FGT: return ARMEmitter::Condition::CC_GT;
case FEXCore::IR::COND_FU: return ARMEmitter::Condition::CC_VS;
case FEXCore::IR::COND_FNU: return ARMEmitter::Condition::CC_VC;
case FEXCore::IR::COND_VS:
case FEXCore::IR::COND_VC:
case FEXCore::IR::COND_MI: return ARMEmitter::Condition::CC_MI;
case FEXCore::IR::COND_PL: return ARMEmitter::Condition::CC_PL;
default: LOGMAN_MSG_A_FMT("Unsupported compare type"); return ARMEmitter::Condition::CC_NV;
}
}
[[nodiscard]]
bool IsFPR(IR::NodeID Node) const;
[[nodiscard]]
bool IsGPR(IR::NodeID Node) const;
[[nodiscard]]
bool IsGPRPair(IR::NodeID Node) const;
[[nodiscard]]
ARMEmitter::ExtendedMemOperand GenerateMemOperand(uint8_t AccessSize, ARMEmitter::Register Base, IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType, uint8_t OffsetScale);
// NOTE: Will use TMP1 as a way to encode immediates that happen to fall outside
// the limits of the scalar plus immediate variant of SVE load/stores.
//
// TMP1 is safe to use again once this memory operand is used with its
// equivalent loads or stores that this was called for.
[[nodiscard]] FEXCore::ARMEmitter::SVEMemOperand GenerateSVEMemOperand(uint8_t AccessSize,
FEXCore::ARMEmitter::Register Base,
IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType,
uint8_t OffsetScale);
[[nodiscard]]
ARMEmitter::SVEMemOperand GenerateSVEMemOperand(uint8_t AccessSize, ARMEmitter::Register Base, IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType, uint8_t OffsetScale);
[[nodiscard]] bool IsInlineConstant(const IR::OrderedNodeWrapper& Node, uint64_t* Value = nullptr) const;
[[nodiscard]] bool IsInlineEntrypointOffset(const IR::OrderedNodeWrapper& WNode, uint64_t* Value) const;
[[nodiscard]]
bool IsInlineConstant(const IR::OrderedNodeWrapper& Node, uint64_t* Value = nullptr) const;
[[nodiscard]]
bool IsInlineEntrypointOffset(const IR::OrderedNodeWrapper& WNode, uint64_t* Value) const;
struct LiveRange {
uint32_t Begin;
@@ -144,79 +281,87 @@ private:
// This is purely a debugging aid for developers to see if they are in JIT code space when inspecting raw memory
void EmitDetectionString();
IR::RegisterAllocationPass *RAPass;
IR::RegisterAllocationData *RAData;
FEXCore::Core::DebugData *DebugData;
IR::RegisterAllocationPass* RAPass;
const IR::RegisterAllocationData* RAData;
FEXCore::Core::DebugData* DebugData;
void ResetStack();
/**
* @name Relocations
* @{ */
uint64_t GetNamedSymbolLiteral(FEXCore::CPU::RelocNamedSymbolLiteral::NamedSymbol Op);
uint64_t GetNamedSymbolLiteral(FEXCore::CPU::RelocNamedSymbolLiteral::NamedSymbol Op);
/**
* @brief A literal pair relocation object for named symbol literals
*/
struct NamedSymbolLiteralPair {
ARMEmitter::ForwardLabel Loc;
uint64_t Lit;
Relocation MoveABI{};
};
/**
* @brief A literal pair relocation object for named symbol literals
*/
struct NamedSymbolLiteralPair {
ARMEmitter::ForwardLabel Loc;
uint64_t Lit;
Relocation MoveABI {};
};
/**
* @brief Inserts a thunk relocation
*
* @param Reg - The GPR to move the thunk handler in to
* @param Sum - The hash of the thunk
*/
void InsertNamedThunkRelocation(ARMEmitter::Register Reg, const IR::SHA256Sum &Sum);
/**
* @brief Inserts a thunk relocation
*
* @param Reg - The GPR to move the thunk handler in to
* @param Sum - The hash of the thunk
*/
void InsertNamedThunkRelocation(ARMEmitter::Register Reg, const IR::SHA256Sum& Sum);
/**
* @brief Inserts a guest GPR move relocation
*
* @param Reg - The GPR to move the guest RIP in to
* @param Constant - The guest RIP that will be relocated
*/
void InsertGuestRIPMove(ARMEmitter::Register Reg, uint64_t Constant);
/**
* @brief Inserts a guest GPR move relocation
*
* @param Reg - The GPR to move the guest RIP in to
* @param Constant - The guest RIP that will be relocated
*/
void InsertGuestRIPMove(ARMEmitter::Register Reg, uint64_t Constant);
/**
* @brief Inserts a named symbol as a literal in memory
*
* Need to use `PlaceNamedSymbolLiteral` with the return value to place the literal in the desired location
*
* @param Op The named symbol to place
*
* @return A temporary `NamedSymbolLiteralPair`
*/
NamedSymbolLiteralPair InsertNamedSymbolLiteral(FEXCore::CPU::RelocNamedSymbolLiteral::NamedSymbol Op);
/**
* @brief Inserts a named symbol as a literal in memory
*
* Need to use `PlaceNamedSymbolLiteral` with the return value to place the literal in the desired location
*
* @param Op The named symbol to place
*
* @return A temporary `NamedSymbolLiteralPair`
*/
NamedSymbolLiteralPair InsertNamedSymbolLiteral(FEXCore::CPU::RelocNamedSymbolLiteral::NamedSymbol Op);
/**
* @brief Place the named symbol literal relocation in memory
*
* @param Lit - Which literal to place
*/
void PlaceNamedSymbolLiteral(NamedSymbolLiteralPair &Lit);
/**
* @brief Place the named symbol literal relocation in memory
*
* @param Lit - Which literal to place
*/
void PlaceNamedSymbolLiteral(NamedSymbolLiteralPair& Lit);
fextl::vector<FEXCore::CPU::Relocation> Relocations;
fextl::vector<FEXCore::CPU::Relocation> Relocations;
///< Relocation code loading
bool ApplyRelocations(uint64_t GuestEntry, uint64_t CodeEntry, uint64_t CursorEntry, size_t NumRelocations, const char* EntryRelocations);
///< Relocation code loading
bool ApplyRelocations(uint64_t GuestEntry, uint64_t CodeEntry, uint64_t CursorEntry, size_t NumRelocations, const char* EntryRelocations);
/** @} */
uint32_t SpillSlots{};
using OpType = void (Arm64JITCore::*)(IR::IROp_Header const *IROp, IR::NodeID Node);
uint32_t SpillSlots {};
using OpType = void (Arm64JITCore::*)(const IR::IROp_Header* IROp, IR::NodeID Node);
using ScalarBinaryOpCaller = std::function<void(ARMEmitter::VRegister Dst, ARMEmitter::VRegister Src1, ARMEmitter::VRegister Src2)>;
void VFScalarOperation(uint8_t OpSize, uint8_t ElementSize, bool ZeroUpperBits, ScalarBinaryOpCaller ScalarEmit,
ARMEmitter::VRegister Dst, ARMEmitter::VRegister Vector1, ARMEmitter::VRegister Vector2);
using ScalarUnaryOpCaller = std::function<void(ARMEmitter::VRegister Dst, std::variant<ARMEmitter::VRegister, ARMEmitter::Register> SrcVar)>;
void VFScalarUnaryOperation(uint8_t OpSize, uint8_t ElementSize, bool ZeroUpperBits, ScalarUnaryOpCaller ScalarEmit, ARMEmitter::VRegister Dst,
ARMEmitter::VRegister Vector1, std::variant<ARMEmitter::VRegister, ARMEmitter::Register> Vector2);
// Runtime selection;
// Load and store register style.
OpType RT_LoadRegister;
OpType RT_StoreRegister;
// Load and store TSO memory style
OpType RT_LoadMemTSO;
OpType RT_StoreMemTSO;
#define DEF_OP(x) void Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
// Dynamic Dispatcher supporting operations
DEF_OP(ParanoidLoadMemTSO);
DEF_OP(ParanoidStoreMemTSO);
///< Unhandled handler
DEF_OP(Unhandled);
@@ -224,272 +369,8 @@ private:
///< No-op Handler
DEF_OP(NoOp);
///< ALU Ops
DEF_OP(TruncElementPair);
DEF_OP(Constant);
DEF_OP(EntrypointOffset);
DEF_OP(InlineConstant);
DEF_OP(InlineEntrypointOffset);
DEF_OP(CycleCounter);
DEF_OP(Add);
DEF_OP(AddNZCV);
DEF_OP(AdcNZCV);
DEF_OP(SbbNZCV);
DEF_OP(TestNZ);
DEF_OP(Sub);
DEF_OP(SubNZCV);
DEF_OP(Neg);
DEF_OP(Abs);
DEF_OP(Mul);
DEF_OP(UMul);
DEF_OP(Div);
DEF_OP(UDiv);
DEF_OP(Rem);
DEF_OP(URem);
DEF_OP(MulH);
DEF_OP(UMulH);
DEF_OP(Or);
DEF_OP(Orlshl);
DEF_OP(Orlshr);
DEF_OP(Ornror);
DEF_OP(And);
DEF_OP(Andn);
DEF_OP(Xor);
DEF_OP(Lshl);
DEF_OP(Lshr);
DEF_OP(Ashr);
DEF_OP(Rol);
DEF_OP(Ror);
DEF_OP(Extr);
DEF_OP(PDep);
DEF_OP(PExt);
DEF_OP(LDiv);
DEF_OP(LUDiv);
DEF_OP(LRem);
DEF_OP(LURem);
DEF_OP(Zext);
DEF_OP(Not);
DEF_OP(Popcount);
DEF_OP(FindLSB);
DEF_OP(FindMSB);
DEF_OP(FindTrailingZeroes);
DEF_OP(CountLeadingZeroes);
DEF_OP(Rev);
DEF_OP(Bfi);
DEF_OP(Bfxil);
DEF_OP(Bfe);
DEF_OP(Sbfe);
DEF_OP(Select);
DEF_OP(VExtractToGPR);
DEF_OP(Float_ToGPR_ZU);
DEF_OP(Float_ToGPR_ZS);
DEF_OP(Float_ToGPR_S);
DEF_OP(FCmp);
///< Atomic ops
DEF_OP(CASPair);
DEF_OP(CAS);
DEF_OP(AtomicAdd);
DEF_OP(AtomicSub);
DEF_OP(AtomicAnd);
DEF_OP(AtomicOr);
DEF_OP(AtomicXor);
DEF_OP(AtomicSwap);
DEF_OP(AtomicFetchAdd);
DEF_OP(AtomicFetchSub);
DEF_OP(AtomicFetchAnd);
DEF_OP(AtomicFetchCLR);
DEF_OP(AtomicFetchOr);
DEF_OP(AtomicFetchXor);
DEF_OP(AtomicFetchNeg);
DEF_OP(TelemetrySetValue);
///< Branch ops
DEF_OP(CallbackReturn);
DEF_OP(ExitFunction);
DEF_OP(Jump);
DEF_OP(CondJump);
DEF_OP(Syscall);
DEF_OP(InlineSyscall);
DEF_OP(Thunk);
DEF_OP(ValidateCode);
DEF_OP(ThreadRemoveCodeEntry);
DEF_OP(CPUID);
DEF_OP(XGETBV);
///< Conversion ops
DEF_OP(VInsGPR);
DEF_OP(VCastFromGPR);
DEF_OP(VDupFromGPR);
DEF_OP(Float_FromGPR_S);
DEF_OP(Float_FToF);
DEF_OP(Vector_SToF);
DEF_OP(Vector_FToZS);
DEF_OP(Vector_FToS);
DEF_OP(Vector_FToF);
DEF_OP(Vector_FToI);
///< Flag ops
DEF_OP(GetHostFlag);
///< Memory ops
DEF_OP(LoadContext);
DEF_OP(StoreContext);
DEF_OP(LoadRegister);
DEF_OP(StoreRegister);
DEF_OP(LoadRegisterSRA);
DEF_OP(StoreRegisterSRA);
DEF_OP(LoadContextIndexed);
DEF_OP(StoreContextIndexed);
DEF_OP(SpillRegister);
DEF_OP(FillRegister);
DEF_OP(LoadFlag);
DEF_OP(StoreFlag);
DEF_OP(LoadMem);
DEF_OP(StoreMem);
DEF_OP(LoadMemTSO);
DEF_OP(StoreMemTSO);
DEF_OP(VLoadVectorMasked);
DEF_OP(VStoreVectorMasked);
DEF_OP(VLoadVectorElement);
DEF_OP(VStoreVectorElement);
DEF_OP(VBroadcastFromMem);
DEF_OP(Push);
DEF_OP(MemSet);
DEF_OP(MemCpy);
DEF_OP(ParanoidLoadMemTSO);
DEF_OP(ParanoidStoreMemTSO);
DEF_OP(CacheLineClear);
DEF_OP(CacheLineClean);
DEF_OP(CacheLineZero);
///< Misc ops
DEF_OP(GuestOpcode);
DEF_OP(Fence);
DEF_OP(Break);
DEF_OP(Print);
DEF_OP(GetRoundingMode);
DEF_OP(SetRoundingMode);
DEF_OP(ProcessorID);
DEF_OP(RDRAND);
DEF_OP(Yield);
///< Move ops
DEF_OP(ExtractElementPair);
DEF_OP(CreateElementPair);
///< Vector ops
DEF_OP(VectorZero);
DEF_OP(VectorImm);
DEF_OP(LoadNamedVectorConstant);
DEF_OP(LoadNamedVectorIndexedConstant);
DEF_OP(VMov);
DEF_OP(VAnd);
DEF_OP(VBic);
DEF_OP(VOr);
DEF_OP(VXor);
DEF_OP(VAdd);
DEF_OP(VSub);
DEF_OP(VUQAdd);
DEF_OP(VUQSub);
DEF_OP(VSQAdd);
DEF_OP(VSQSub);
DEF_OP(VAddP);
DEF_OP(VAddV);
DEF_OP(VUMinV);
DEF_OP(VURAvg);
DEF_OP(VAbs);
DEF_OP(VFAbs);
DEF_OP(VPopcount);
DEF_OP(VFAdd);
DEF_OP(VFAddP);
DEF_OP(VFSub);
DEF_OP(VFMul);
DEF_OP(VFDiv);
DEF_OP(VFMin);
DEF_OP(VFMax);
DEF_OP(VFRecp);
DEF_OP(VFSqrt);
DEF_OP(VFRSqrt);
DEF_OP(VNeg);
DEF_OP(VFNeg);
DEF_OP(VNot);
DEF_OP(VUMin);
DEF_OP(VSMin);
DEF_OP(VUMax);
DEF_OP(VSMax);
DEF_OP(VZip);
DEF_OP(VZip2);
DEF_OP(VUnZip);
DEF_OP(VUnZip2);
DEF_OP(VTrn);
DEF_OP(VTrn2);
DEF_OP(VBSL);
DEF_OP(VCMPEQ);
DEF_OP(VCMPEQZ);
DEF_OP(VCMPGT);
DEF_OP(VCMPGTZ);
DEF_OP(VCMPLTZ);
DEF_OP(VFCMPEQ);
DEF_OP(VFCMPNEQ);
DEF_OP(VFCMPLT);
DEF_OP(VFCMPGT);
DEF_OP(VFCMPLE);
DEF_OP(VFCMPORD);
DEF_OP(VFCMPUNO);
DEF_OP(VUShl);
DEF_OP(VUShr);
DEF_OP(VSShr);
DEF_OP(VUShlS);
DEF_OP(VUShrS);
DEF_OP(VSShrS);
DEF_OP(VUShrSWide);
DEF_OP(VSShrSWide);
DEF_OP(VUShlSWide);
DEF_OP(VInsElement);
DEF_OP(VDupElement);
DEF_OP(VExtr);
DEF_OP(VUShrI);
DEF_OP(VSShrI);
DEF_OP(VShlI);
DEF_OP(VUShrNI);
DEF_OP(VUShrNI2);
DEF_OP(VSXTL);
DEF_OP(VSXTL2);
DEF_OP(VUXTL);
DEF_OP(VUXTL2);
DEF_OP(VSQXTN);
DEF_OP(VSQXTN2);
DEF_OP(VSQXTNPair);
DEF_OP(VSQXTUN);
DEF_OP(VSQXTUN2);
DEF_OP(VSQXTUNPair);
DEF_OP(VSRSHR);
DEF_OP(VSQSHL);
DEF_OP(VMul);
DEF_OP(VUMull);
DEF_OP(VSMull);
DEF_OP(VUMull2);
DEF_OP(VSMull2);
DEF_OP(VUMulH);
DEF_OP(VSMulH);
DEF_OP(VUABDL);
DEF_OP(VUABDL2);
DEF_OP(VTBL1);
DEF_OP(VTBL2);
DEF_OP(VRev32);
DEF_OP(VRev64);
DEF_OP(VFCADD);
///< Encryption ops
DEF_OP(AESImc);
DEF_OP(AESEnc);
DEF_OP(AESEncLast);
DEF_OP(AESDec);
DEF_OP(AESDecLast);
DEF_OP(AESKeyGenAssist);
DEF_OP(CRC32);
DEF_OP(PCLMUL);
#define IROP_DISPATCH_DEFS
#include <FEXCore/IR/IRDefines_Dispatch.inc>
#undef DEF_OP
};
File diff suppressed because it is too large. Load diff
@@ -10,14 +10,13 @@ $end_info$
#endif
#include "Interface/Context/Context.h"
#include "Interface/Core/ArchHelpers/CodeEmitter/Emitter.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "FEXCore/Debug/InternalThreadState.h"
#include <FEXCore/Core/SignalDelegator.h>
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(GuestOpcode) {
auto Op = IROp->C<IR::IROp_GuestOpcode>();
@@ -28,16 +27,10 @@ DEF_OP(GuestOpcode) {
DEF_OP(Fence) {
auto Op = IROp->C<IR::IROp_Fence>();
switch (Op->Fence) {
case IR::Fence_Load.Val:
dmb(FEXCore::ARMEmitter::BarrierScope::LD);
break;
case IR::Fence_LoadStore.Val:
dmb(FEXCore::ARMEmitter::BarrierScope::SY);
break;
case IR::Fence_Store.Val:
dmb(FEXCore::ARMEmitter::BarrierScope::ST);
break;
default: LOGMAN_MSG_A_FMT("Unknown Fence: {}", Op->Fence); break;
case IR::Fence_Load.Val: dmb(ARMEmitter::BarrierScope::LD); break;
case IR::Fence_LoadStore.Val: dmb(ARMEmitter::BarrierScope::SY); break;
case IR::Fence_Store.Val: dmb(ARMEmitter::BarrierScope::ST); break;
default: LOGMAN_MSG_A_FMT("Unknown Fence: {}", Op->Fence); break;
}
}
@@ -55,7 +48,7 @@ DEF_OP(Break) {
.err_code = Op->Reason.ErrorRegister,
};
uint64_t Constant{};
uint64_t Constant {};
memcpy(&Constant, &State, sizeof(State));
LoadConstant(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, Constant);
@@ -127,6 +120,39 @@ DEF_OP(SetRoundingMode) {
msr(ARMEmitter::SystemRegister::FPCR, TMP1);
}
DEF_OP(PushRoundingMode) {
auto Op = IROp->C<IR::IROp_PushRoundingMode>();
auto Dest = GetReg(Node);
// Save the old rounding mode
mrs(Dest, ARMEmitter::SystemRegister::FPCR);
// vixl simulator doesn't support anything beyond ties-to-even rounding
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
return;
}
// Insert the rounding flags, reversing the mode bits as above
if (Op->RoundMode == 3) {
orr(ARMEmitter::Size::i64Bit, TMP1, Dest, 3 << 22);
} else if (Op->RoundMode == 0) {
and_(ARMEmitter::Size::i64Bit, TMP1, Dest, ~(3 << 22));
} else {
LOGMAN_THROW_AA_FMT(Op->RoundMode == 1 || Op->RoundMode == 2, "expect a valid round mode");
and_(ARMEmitter::Size::i64Bit, TMP1, Dest, ~(Op->RoundMode << 22));
orr(ARMEmitter::Size::i64Bit, TMP1, TMP1, (Op->RoundMode == 2 ? 1 : 2) << 22);
}
// Now save the new FPCR
msr(ARMEmitter::SystemRegister::FPCR, TMP1);
}
DEF_OP(PopRoundingMode) {
auto Op = IROp->C<IR::IROp_PopRoundingMode>();
msr(ARMEmitter::SystemRegister::FPCR, GetReg(Op->FPCR.ID()));
}
DEF_OP(Print) {
auto Op = IROp->C<IR::IROp_Print>();
@@ -136,14 +162,21 @@ DEF_OP(Print) {
if (IsGPR(Op->Value.ID())) {
mov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, GetReg(Op->Value.ID()));
ldr(ARMEmitter::XReg::x3, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.PrintValue));
}
else {
} else {
fmov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r0, GetVReg(Op->Value.ID()), false);
fmov(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::r1, GetVReg(Op->Value.ID()), true);
ldr(ARMEmitter::XReg::x3, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.PrintVectorValue));
}
blr(ARMEmitter::Reg::r3);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
if (IsGPR(Op->Value.ID())) {
GenerateIndirectRuntimeCall<void, uint64_t>(ARMEmitter::Reg::r3);
} else {
GenerateIndirectRuntimeCall<void, uint64_t, uint64_t>(ARMEmitter::Reg::r3);
}
} else {
blr(ARMEmitter::Reg::r3);
}
FillStaticRegs();
PopDynamicRegsAndLR();
@@ -221,8 +254,7 @@ DEF_OP(RDRAND) {
if (Op->GetReseeded) {
mrs(Dst.first, ARMEmitter::SystemRegister::RNDRRS);
}
else {
} else {
mrs(Dst.first, ARMEmitter::SystemRegister::RNDR);
}
@@ -235,5 +267,4 @@ DEF_OP(Yield) {
}
#undef DEF_OP
}
} // namespace FEXCore::CPU
@@ -8,15 +8,17 @@ $end_info$
#include "Interface/Core/JIT/Arm64/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const *IROp, IR::NodeID Node)
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(ExtractElementPair) {
auto Op = IROp->C<IR::IROp_ExtractElementPair>();
LOGMAN_THROW_AA_FMT(Op->Header.Size == 4 || Op->Header.Size == 8, "Invalid size");
const auto EmitSize = Op->Header.Size == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto Src = GetRegPair(Op->Pair.ID());
const std::array<ARMEmitter::Register, 2> Regs = {Src.first, Src.second};
mov(EmitSize, GetReg(Node), Regs[Op->Element]);
const auto Dst = GetReg(Node);
const auto Pair = GetRegPair(Op->Pair.ID());
const auto Src = Op->Element == 0 ? Pair.first : Pair.second;
if (Dst != Src) {
mov(ConvertSize48(IROp), Dst, Src);
}
}
DEF_OP(CreateElementPair) {
@@ -42,6 +44,25 @@ DEF_OP(CreateElementPair) {
}
}
#undef DEF_OP
DEF_OP(Copy) {
auto Op = IROp->C<IR::IROp_Copy>();
mov(ARMEmitter::Size::i64Bit, GetReg(Node), GetReg(Op->Source.ID()));
}
DEF_OP(Swap1) {
auto Op = IROp->C<IR::IROp_Swap1>();
auto A = GetReg(Op->A.ID()), B = GetReg(Op->B.ID());
LOGMAN_THROW_AA_FMT(B == GetReg(Node), "Invariant");
mov(ARMEmitter::Size::i64Bit, TMP1, A);
mov(ARMEmitter::Size::i64Bit, A, B);
mov(ARMEmitter::Size::i64Bit, B, TMP1);
}
DEF_OP(Swap2) {
// Implemented above
}
#undef DEF_OP
} // namespace FEXCore::CPU
File diff suppressed because it is too large. Load diff
+3 -7
View File
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <FEXCore/Core/CPUBackend.h>
#include "Interface/Core/CPUBackend.h"
#include <FEXCore/fextl/memory.h>
namespace FEXCore::Context {
@@ -15,12 +15,8 @@ struct InternalThreadState;
namespace FEXCore::CPU {
class CPUBackend;
[[nodiscard]] fextl::unique_ptr<CPUBackend> CreateX86JITCore(FEXCore::Context::ContextImpl *ctx,
FEXCore::Core::InternalThreadState *Thread);
CPUBackendFeatures GetX86JITBackendFeatures();
[[nodiscard]] fextl::unique_ptr<CPUBackend> CreateArm64JITCore(FEXCore::Context::ContextImpl *ctx,
FEXCore::Core::InternalThreadState *Thread);
[[nodiscard]]
fextl::unique_ptr<CPUBackend> CreateArm64JITCore(FEXCore::Context::ContextImpl* ctx, FEXCore::Core::InternalThreadState* Thread);
CPUBackendFeatures GetArm64JITBackendFeatures();
} // namespace FEXCore::CPU
@@ -13,8 +13,8 @@ $end_info$
#include "Interface/Core/LookupCache.h"
namespace FEXCore {
LookupCache::LookupCache(FEXCore::Context::ContextImpl *CTX)
: BlockLinks_mbr { fextl::pmr::get_default_resource() }
LookupCache::LookupCache(FEXCore::Context::ContextImpl* CTX)
: BlockLinks_mbr {fextl::pmr::get_default_resource()}
, ctx {CTX} {
TotalCacheSize = ctx->Config.VirtualMemSize / 4096 * 8 + CODE_SIZE + L1_SIZE;
@@ -78,5 +78,4 @@ void LookupCache::ClearCache() {
BlockList.clear();
}
}
} // namespace FEXCore
+61 -35
View File
@@ -13,6 +13,9 @@
#include <stddef.h>
#include <utility>
#include <mutex>
#ifdef _M_ARM_64EC
#include <winnt.h>
#endif
namespace FEXCore {
@@ -23,12 +26,12 @@ public:
uintptr_t GuestCode;
};
LookupCache(FEXCore::Context::ContextImpl *CTX);
LookupCache(FEXCore::Context::ContextImpl* CTX);
~LookupCache();
uintptr_t FindBlock(uint64_t Address) {
// Try L1, no lock needed
auto &L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
auto& L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
if (L1Entry.GuestCode == Address) {
return L1Entry.HostCode;
}
@@ -37,7 +40,7 @@ public:
std::lock_guard<std::recursive_mutex> lk(WriteLock);
// Try L2
const auto PageIndex = (Address & (VirtualMemSize -1)) >> 12;
const auto PageIndex = (Address & (VirtualMemSize - 1)) >> 12;
const auto PageOffset = Address & (0x0FFF);
const auto Pointers = reinterpret_cast<uintptr_t*>(PagePointer);
@@ -48,8 +51,7 @@ public:
// Find there pointer for the address in the blocks
auto BlockPointers = reinterpret_cast<LookupCacheEntry*>(LocalPagePointer);
if (BlockPointers[PageOffset].GuestCode == Address)
{
if (BlockPointers[PageOffset].GuestCode == Address) {
L1Entry.GuestCode = Address;
L1Entry.HostCode = BlockPointers[PageOffset].HostCode;
return L1Entry.HostCode;
@@ -68,6 +70,24 @@ public:
return 0;
}
#ifdef _M_ARM_64EC
bool CheckPageEC(uint64_t Address) {
if (!RtlIsEcCode(Address)) {
return false;
}
std::lock_guard<std::recursive_mutex> lk(WriteLock);
// Mark L2 entry for this page as EC by setting the LSB, this can then be
// checked by the dispatcher to see if it needs to perform a call/return to
// EC code.
const auto PageIndex = (Address & (VirtualMemSize - 1)) >> 12;
const auto Pointers = reinterpret_cast<uintptr_t*>(PagePointer);
Pointers[PageIndex] |= 1;
return true;
}
#endif
fextl::map<uint64_t, fextl::vector<uint64_t>> CodePages;
// Appends Block {Address} to CodePages [Start, Start + Length)
@@ -77,8 +97,8 @@ public:
bool rv = false;
for (auto CurrentPage = Start >> 12, EndPage = (Start + Length -1) >> 12; CurrentPage <= EndPage; CurrentPage++) {
auto &CodePage = CodePages[CurrentPage];
for (auto CurrentPage = Start >> 12, EndPage = (Start + Length - 1) >> 12; CurrentPage <= EndPage; CurrentPage++) {
auto& CodePage = CodePages[CurrentPage];
rv |= CodePage.size() == 0;
CodePage.push_back(Address);
}
@@ -87,7 +107,7 @@ public:
}
// Adds to Guest -> Host code mapping
void AddBlockMapping(uint64_t Address, void *HostCode) {
void AddBlockMapping(uint64_t Address, void* HostCode) {
std::lock_guard<std::recursive_mutex> lk(WriteLock);
[[maybe_unused]] auto Inserted = BlockList.emplace(Address, (uintptr_t)HostCode).second;
@@ -95,27 +115,27 @@ public:
// There is no need to update L1 or L2, they will get updated on first lookup
// However, adding to L1 here increases performance
auto &L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
auto& L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
L1Entry.GuestCode = Address;
L1Entry.HostCode = (uintptr_t)HostCode;
}
void Erase(uint64_t Address) {
void Erase(FEXCore::Core::CpuStateFrame* Frame, uint64_t Address) {
std::lock_guard<std::recursive_mutex> lk(WriteLock);
// Sever any links to this block
auto lower = BlockLinks->lower_bound({Address, 0});
auto upper = BlockLinks->upper_bound({Address, UINTPTR_MAX});
auto lower = BlockLinks->lower_bound({Address, nullptr});
auto upper = BlockLinks->upper_bound({Address, reinterpret_cast<FEXCore::Context::ExitFunctionLinkData*>(UINTPTR_MAX)});
for (auto it = lower; it != upper; it = BlockLinks->erase(it)) {
it->second();
it->second(Frame, it->first.HostLink);
}
// Remove from BlockList
BlockList.erase(Address);
// Do L1
auto &L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
auto& L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
if (L1Entry.GuestCode == Address) {
L1Entry.GuestCode = 0;
// Leave L1Entry.HostCode as is, so that concurrent lookups won't read a null pointer
@@ -124,11 +144,11 @@ public:
}
// Do full map
Address = Address & (VirtualMemSize -1);
Address = Address & (VirtualMemSize - 1);
uint64_t PageOffset = Address & (0x0FFF);
Address >>= 12;
uintptr_t *Pointers = reinterpret_cast<uintptr_t*>(PagePointer);
uintptr_t* Pointers = reinterpret_cast<uintptr_t*>(PagePointer);
uint64_t LocalPagePointer = Pointers[Address];
if (!LocalPagePointer) {
// Page for this code didn't even exist, nothing to do
@@ -141,8 +161,7 @@ public:
BlockPointers[PageOffset].HostCode = 0;
}
void AddBlockLink(uint64_t GuestDestination, uintptr_t HostLink, const std::function<void()> &delinker) {
void AddBlockLink(uint64_t GuestDestination, FEXCore::Context::ExitFunctionLinkData* HostLink, const FEXCore::Context::BlockDelinkerFunc& delinker) {
std::lock_guard<std::recursive_mutex> lk(WriteLock);
BlockLinks->insert({{GuestDestination, HostLink}, delinker});
@@ -151,14 +170,20 @@ public:
void ClearCache();
void ClearL2Cache();
uintptr_t GetL1Pointer() const { return L1Pointer; }
uintptr_t GetPagePointer() const { return PagePointer; }
uintptr_t GetVirtualMemorySize() const { return VirtualMemSize; }
uintptr_t GetL1Pointer() const {
return L1Pointer;
}
uintptr_t GetPagePointer() const {
return PagePointer;
}
uintptr_t GetVirtualMemorySize() const {
return VirtualMemSize;
}
constexpr static size_t L1_ENTRIES = 1 * 1024 * 1024; // Must be a power of 2
constexpr static size_t L1_ENTRIES_MASK = L1_ENTRIES - 1;
// This needs to be taken before reads or writes to L2, L3, CodePages, Thread::DebugStore,
// This needs to be taken before reads or writes to L2, L3, CodePages,
// and before writes to L1. Concurrent access from a thread that this LookupCache doesn't belong to
// may only happen during cross thread invalidation (::Erase).
// All other operations must be done from the owning thread.
@@ -170,17 +195,17 @@ public:
private:
void CacheBlockMapping(uint64_t Address, uintptr_t HostCode) {
// Do L1
auto &L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
auto& L1Entry = reinterpret_cast<LookupCacheEntry*>(L1Pointer)[Address & L1_ENTRIES_MASK];
L1Entry.GuestCode = Address;
L1Entry.HostCode = HostCode;
// Do ful map
auto FullAddress = Address;
Address = Address & (VirtualMemSize -1);
Address = Address & (VirtualMemSize - 1);
uint64_t PageOffset = Address & (0x0FFF);
Address >>= 12;
uintptr_t *Pointers = reinterpret_cast<uintptr_t*>(PagePointer);
uintptr_t* Pointers = reinterpret_cast<uintptr_t*>(PagePointer);
uint64_t LocalPagePointer = Pointers[Address];
if (!LocalPagePointer) {
// We don't have a page pointer for this address
@@ -224,15 +249,16 @@ private:
struct BlockLinkTag {
uint64_t GuestDestination;
uintptr_t HostLink;
FEXCore::Context::ExitFunctionLinkData* HostLink;
bool operator <(const BlockLinkTag& other) const {
if (GuestDestination < other.GuestDestination)
bool operator<(const BlockLinkTag& other) const {
if (GuestDestination < other.GuestDestination) {
return true;
else if (GuestDestination == other.GuestDestination)
} else if (GuestDestination == other.GuestDestination) {
return HostLink < other.HostLink;
else
} else {
return false;
}
}
};
@@ -243,9 +269,9 @@ private:
//
// This makes `BlockLinks` look like a raw pointer that could memory leak, but since it is backed by the MBR, it won't.
std::pmr::monotonic_buffer_resource BlockLinks_mbr;
using BlockLinksMapType = std::pmr::map<BlockLinkTag, std::function<void()>>;
using BlockLinksMapType = std::pmr::map<BlockLinkTag, FEXCore::Context::BlockDelinkerFunc>;
fextl::unique_ptr<std::pmr::polymorphic_allocator<std::byte>> BlockLinks_pma;
BlockLinksMapType *BlockLinks;
BlockLinksMapType* BlockLinks;
fextl::robin_map<uint64_t, uint64_t> BlockList;
@@ -257,7 +283,7 @@ private:
size_t AllocateOffset {};
FEXCore::Context::ContextImpl *ctx;
uint64_t VirtualMemSize{};
FEXCore::Context::ContextImpl* ctx;
uint64_t VirtualMemSize {};
};
}
} // namespace FEXCore
@@ -6,84 +6,81 @@
#include <cstdint>
namespace FEXCore::CodeSerialize {
// If any of the config options mismatch on load then the cache won't be used
// Any of these will result in codegen changes
struct
FEX_PACKED
CodeObjectSerializationConfig {
// Cookie in the header of the file, isn't part of the config hash
uint64_t Cookie{};
// If any of the config options mismatch on load then the cache won't be used
// Any of these will result in codegen changes
struct FEX_PACKED CodeObjectSerializationConfig {
// Cookie in the header of the file, isn't part of the config hash
uint64_t Cookie {};
// Instructions per block configuration
int32_t MaxInstPerBlock{};
// Instructions per block configuration
int32_t MaxInstPerBlock {};
// Follows CPUID 4000_0001_EAX[3:0]
unsigned Arch : 4;
// Follows CPUID 4000_0001_EAX[3:0]
unsigned Arch : 4;
// Multiblock enabled
unsigned MultiBlock : 1;
// Multiblock enabled
unsigned MultiBlock : 1;
// Hardware TSO enabled
unsigned HardwareTSOEnabled : 1;
// Hardware TSO enabled
unsigned HardwareTSOEnabled : 1;
// TSO enabled
unsigned TSOEnabled : 1;
// TSO enabled
unsigned TSOEnabled : 1;
// ABI local flag unsafe optimization
unsigned ABILocalFlags : 1;
// ABI local flag unsafe optimization
unsigned ABILocalFlags : 1;
// Static register allocation enabled
unsigned SRA : 1;
// Paranoid TSO mode enabled
unsigned ParanoidTSO : 1;
// Paranoid TSO mode enabled
unsigned ParanoidTSO : 1;
// Guest code execution mode (We don't support live mode switch)
unsigned Is64BitMode : 1;
// Guest code execution mode (We don't support live mode switch)
unsigned Is64BitMode : 1;
// SMC checks style
unsigned SMCChecks : 2;
// SMC checks style
unsigned SMCChecks : 2;
// x87 reduced precision
unsigned x87ReducedPrecision : 1;
// x87 reduced precision
unsigned x87ReducedPrecision : 1;
// Padding to remove uninitialized data warning from asan
// Shows remaining amount of bits available for config
unsigned _Pad : 19;
// Padding to remove uninitialized data warning from asan
// Shows remaining amount of bits available for config
unsigned _Pad : 18;
bool operator==(const CodeObjectSerializationConfig& other) const {
return Cookie == other.Cookie && MaxInstPerBlock == other.MaxInstPerBlock && Arch == other.Arch && MultiBlock == other.MultiBlock &&
HardwareTSOEnabled == other.HardwareTSOEnabled && TSOEnabled == other.TSOEnabled && ABILocalFlags == other.ABILocalFlags &&
ParanoidTSO == other.ParanoidTSO && Is64BitMode == other.Is64BitMode && SMCChecks == other.SMCChecks &&
x87ReducedPrecision == other.x87ReducedPrecision;
}
static uint64_t GetHash(const CodeObjectSerializationConfig& other) {
// For < 64-bits of data just pack directly
// Skip the cookie
uint64_t Hash {};
Hash <<= 32;
Hash |= other.MaxInstPerBlock;
Hash <<= 1;
Hash |= other.Arch;
Hash <<= 1;
Hash |= other.MultiBlock;
Hash <<= 1;
Hash |= other.HardwareTSOEnabled;
Hash <<= 1;
Hash |= other.TSOEnabled;
Hash <<= 1;
Hash |= other.ABILocalFlags;
Hash <<= 1;
Hash |= other.ParanoidTSO;
Hash <<= 1;
Hash |= other.Is64BitMode;
Hash <<= 2;
Hash |= other.SMCChecks;
Hash <<= 1;
Hash |= other.x87ReducedPrecision;
return Hash;
}
};
bool operator==(CodeObjectSerializationConfig const &other) const {
return Cookie == other.Cookie &&
MaxInstPerBlock == other.MaxInstPerBlock &&
Arch == other.Arch &&
MultiBlock == other.MultiBlock &&
HardwareTSOEnabled == other.HardwareTSOEnabled &&
TSOEnabled == other.TSOEnabled &&
ABILocalFlags == other.ABILocalFlags &&
SRA == other.SRA &&
ParanoidTSO == other.ParanoidTSO &&
Is64BitMode == other.Is64BitMode &&
SMCChecks == other.SMCChecks &&
x87ReducedPrecision == other.x87ReducedPrecision;
}
static uint64_t GetHash(CodeObjectSerializationConfig const &other) {
// For < 64-bits of data just pack directly
// Skip the cookie
uint64_t Hash{};
Hash <<= 32; Hash |= other.MaxInstPerBlock;
Hash <<= 1; Hash |= other.Arch;
Hash <<= 1; Hash |= other.MultiBlock;
Hash <<= 1; Hash |= other.HardwareTSOEnabled;
Hash <<= 1; Hash |= other.TSOEnabled;
Hash <<= 1; Hash |= other.ABILocalFlags;
Hash <<= 1; Hash |= other.SRA;
Hash <<= 1; Hash |= other.ParanoidTSO;
Hash <<= 1; Hash |= other.Is64BitMode;
Hash <<= 2; Hash |= other.SMCChecks;
Hash <<= 1; Hash |= other.x87ReducedPrecision;
return Hash;
}
};
static_assert(sizeof(CodeObjectSerializationConfig) == 16, "Size changed");
static_assert((sizeof(CodeObjectSerializationConfig) - sizeof(uint64_t)) == 8, "Config size exceeded 64its. Need to change how the hash is generated!");
}
static_assert(sizeof(CodeObjectSerializationConfig) == 16, "Size changed");
static_assert((sizeof(CodeObjectSerializationConfig) - sizeof(uint64_t)) == 8, "Config size exceeded 64its. Need to change how the hash is "
"generated!");
} // namespace FEXCore::CodeSerialize
@@ -11,120 +11,112 @@
#include <xxhash.h>
namespace FEXCore::CodeSerialize {
void AsyncJobHandler::AsyncAddNamedRegionJob(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const fextl::string &filename) {
void AsyncJobHandler::AsyncAddNamedRegionJob(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const fextl::string& filename) {
#ifndef _WIN32
// This function adds a named region *JOB* to our named region handler
// This needs to be as fast as possible to keep out of the way of the JIT
// This function adds a named region *JOB* to our named region handler
// This needs to be as fast as possible to keep out of the way of the JIT
const fextl::string BaseFilename = FHU::Filesystem::GetFilename(filename);
const fextl::string BaseFilename = FHU::Filesystem::GetFilename(filename);
if (!BaseFilename.empty()) {
// Create a new entry that once set up will be put in to our section object map
auto Entry = fextl::make_unique<CodeRegionEntry>(
Base,
Size,
Offset,
filename,
NamedRegionHandler->DefaultCodeHeader(Base, Offset)
);
if (!BaseFilename.empty()) {
// Create a new entry that once set up will be put in to our section object map
auto Entry = fextl::make_unique<CodeRegionEntry>(Base, Size, Offset, filename, NamedRegionHandler->DefaultCodeHeader(Base, Offset));
// Lock the job ref counter so we can block anything attempting to use the entry before it is loaded
Entry->NamedJobRefCountMutex.lock();
// Lock the job ref counter so we can block anything attempting to use the entry before it is loaded
Entry->NamedJobRefCountMutex.lock();
CodeRegionMapType::iterator EntryIterator;
{
std::unique_lock lk {CodeObjectCacheService->GetEntryMapMutex()};
auto &EntryMap = CodeObjectCacheService->GetEntryMap();
auto it = EntryMap.emplace(Base, std::move(Entry));
if (!it.second) {
// This happens when an application overwrites a previous region without unmapping what was there
// Lock this entry's Named job reference counter.
// Once this passes then we know that this section has been loaded.
it.first->second->NamedJobRefCountMutex.lock();
// Finalize anything the region needs to do first.
CodeObjectCacheService->DoCodeRegionClosure(it.first->second->Base, it.first->second.get());
// munmap the file that was mapped
FEXCore::Allocator::munmap(it.first->second->CodeData, it.first->second->FileSize);
// Remove this entry from the unrelocated map as well
{
std::unique_lock lk2 {CodeObjectCacheService->GetUnrelocatedEntryMapMutex()};
CodeObjectCacheService->GetUnrelocatedEntryMap().erase(it.first->second->EntryHeader.OriginalBase);
}
// Now overwrite the entry in the map
it = EntryMap.insert_or_assign(Base, std::move(Entry));
EntryIterator = it.first;
}
else {
// No overwrite, just insert
EntryIterator = it.first;
}
}
// Now that this entry has been added to the map, we can insert a load job using the entry iterator.
// This allows us to quickly unblock the JIT thread when it is loading multiple regions and have the async thread
// do the loading for us.
//
// Create the async work queue job now so it can load
NamedRegionHandler->AsyncAddNamedRegionWorkItem(BaseFilename, filename, true, EntryIterator);
// Tell the async thread that it has work to do
CodeObjectCacheService->NotifyWork();
}
#endif
}
void AsyncJobHandler::AsyncRemoveNamedRegionJob(uintptr_t Base, uintptr_t Size) {
#ifndef _WIN32
// Removing a named region through the job system
// We need to find the entry that we are deleting first
fextl::unique_ptr<CodeRegionEntry> EntryPointer;
CodeRegionMapType::iterator EntryIterator;
{
std::unique_lock lk {CodeObjectCacheService->GetEntryMapMutex()};
auto &EntryMap = CodeObjectCacheService->GetEntryMap();
auto it = EntryMap.find(Base);
if (it != EntryMap.end()) {
// Lock the job ref counter since we are erasing it
// Once this passes it will have been loaded
it->second->NamedJobRefCountMutex.lock();
auto& EntryMap = CodeObjectCacheService->GetEntryMap();
// Take the pointer from the map
EntryPointer = std::move(it->second);
auto it = EntryMap.emplace(Base, std::move(Entry));
if (!it.second) {
// This happens when an application overwrites a previous region without unmapping what was there
// We can now unmap the file data
FEXCore::Allocator::munmap(EntryPointer->CodeData, EntryPointer->FileSize);
// Lock this entry's Named job reference counter.
// Once this passes then we know that this section has been loaded.
it.first->second->NamedJobRefCountMutex.lock();
// Remove this from the entry map
EntryMap.erase(it);
// Finalize anything the region needs to do first.
CodeObjectCacheService->DoCodeRegionClosure(it.first->second->Base, it.first->second.get());
// munmap the file that was mapped
FEXCore::Allocator::munmap(it.first->second->CodeData, it.first->second->FileSize);
// Remove this entry from the unrelocated map as well
{
std::unique_lock lk2 {CodeObjectCacheService->GetUnrelocatedEntryMapMutex()};
CodeObjectCacheService->GetUnrelocatedEntryMap().erase(EntryPointer->EntryHeader.OriginalBase);
CodeObjectCacheService->GetUnrelocatedEntryMap().erase(it.first->second->EntryHeader.OriginalBase);
}
}
else {
// Tried to remove something that wasn't in our code object tracking
return;
}
// Create the async work queue job now so it can finalize what it needs to do
NamedRegionHandler->AsyncRemoveNamedRegionWorkItem(Base, Size, std::move(EntryPointer));
// Tell the async thread that it has work to do
CodeObjectCacheService->NotifyWork();
// Now overwrite the entry in the map
it = EntryMap.insert_or_assign(Base, std::move(Entry));
EntryIterator = it.first;
} else {
// No overwrite, just insert
EntryIterator = it.first;
}
}
#endif
}
void AsyncJobHandler::AsyncAddSerializationJob(fextl::unique_ptr<SerializationJobData> Data) {
// XXX: Actually add serialization job
// Now that this entry has been added to the map, we can insert a load job using the entry iterator.
// This allows us to quickly unblock the JIT thread when it is loading multiple regions and have the async thread
// do the loading for us.
//
// Create the async work queue job now so it can load
NamedRegionHandler->AsyncAddNamedRegionWorkItem(BaseFilename, filename, true, EntryIterator);
// Tell the async thread that it has work to do
CodeObjectCacheService->NotifyWork();
}
#endif
}
void AsyncJobHandler::AsyncRemoveNamedRegionJob(uintptr_t Base, uintptr_t Size) {
#ifndef _WIN32
// Removing a named region through the job system
// We need to find the entry that we are deleting first
fextl::unique_ptr<CodeRegionEntry> EntryPointer;
{
std::unique_lock lk {CodeObjectCacheService->GetEntryMapMutex()};
auto& EntryMap = CodeObjectCacheService->GetEntryMap();
auto it = EntryMap.find(Base);
if (it != EntryMap.end()) {
// Lock the job ref counter since we are erasing it
// Once this passes it will have been loaded
it->second->NamedJobRefCountMutex.lock();
// Take the pointer from the map
EntryPointer = std::move(it->second);
// We can now unmap the file data
FEXCore::Allocator::munmap(EntryPointer->CodeData, EntryPointer->FileSize);
// Remove this from the entry map
EntryMap.erase(it);
// Remove this entry from the unrelocated map as well
{
std::unique_lock lk2 {CodeObjectCacheService->GetUnrelocatedEntryMapMutex()};
CodeObjectCacheService->GetUnrelocatedEntryMap().erase(EntryPointer->EntryHeader.OriginalBase);
}
} else {
// Tried to remove something that wasn't in our code object tracking
return;
}
// Create the async work queue job now so it can finalize what it needs to do
NamedRegionHandler->AsyncRemoveNamedRegionWorkItem(Base, Size, std::move(EntryPointer));
// Tell the async thread that it has work to do
CodeObjectCacheService->NotifyWork();
}
#endif
}
void AsyncJobHandler::AsyncAddSerializationJob(fextl::unique_ptr<SerializationJobData> Data) {
// XXX: Actually add serialization job
}
} // namespace FEXCore::CodeSerialize
@@ -7,67 +7,67 @@
#include <FEXCore/fextl/string.h>
namespace FEXCore::CodeSerialize {
NamedRegionObjectHandler::NamedRegionObjectHandler(FEXCore::Context::ContextImpl *ctx) {
DefaultSerializationConfig.Cookie = CODE_COOKIE;
NamedRegionObjectHandler::NamedRegionObjectHandler(FEXCore::Context::ContextImpl* ctx) {
DefaultSerializationConfig.Cookie = CODE_COOKIE;
// Initialize the Arch from CPUID
uint32_t Arch = ctx->CPUID.RunFunction(0x4000'0001, 0).eax & 0xF;
DefaultSerializationConfig.Arch = Arch;
// Initialize the Arch from CPUID
uint32_t Arch = ctx->CPUID.RunFunction(0x4000'0001, 0).eax & 0xF;
DefaultSerializationConfig.Arch = Arch;
DefaultSerializationConfig.MaxInstPerBlock = ctx->Config.MaxInstPerBlock;
DefaultSerializationConfig.MultiBlock = ctx->Config.Multiblock;
DefaultSerializationConfig.TSOEnabled = ctx->Config.TSOEnabled;
DefaultSerializationConfig.ABILocalFlags = ctx->Config.ABILocalFlags;
DefaultSerializationConfig.SRA = ctx->Config.StaticRegisterAllocation;
DefaultSerializationConfig.ParanoidTSO = ctx->Config.ParanoidTSO;
DefaultSerializationConfig.Is64BitMode = ctx->Config.Is64BitMode;
DefaultSerializationConfig.SMCChecks = ctx->Config.SMCChecks;
DefaultSerializationConfig.x87ReducedPrecision = ctx->Config.x87ReducedPrecision;
}
DefaultSerializationConfig.MaxInstPerBlock = ctx->Config.MaxInstPerBlock;
DefaultSerializationConfig.MultiBlock = ctx->Config.Multiblock;
DefaultSerializationConfig.TSOEnabled = ctx->Config.TSOEnabled;
DefaultSerializationConfig.ABILocalFlags = ctx->Config.ABILocalFlags;
DefaultSerializationConfig.ParanoidTSO = ctx->Config.ParanoidTSO;
DefaultSerializationConfig.Is64BitMode = ctx->Config.Is64BitMode;
DefaultSerializationConfig.SMCChecks = ctx->Config.SMCChecks;
DefaultSerializationConfig.x87ReducedPrecision = ctx->Config.x87ReducedPrecision;
}
void NamedRegionObjectHandler::AddNamedRegionObject(CodeRegionMapType::iterator Entry, const fextl::string &base_filename, const fextl::string &filename, bool Executable) {
// XXX: Add named region objects
void NamedRegionObjectHandler::AddNamedRegionObject(CodeRegionMapType::iterator Entry, const fextl::string& base_filename,
const fextl::string& filename, bool Executable) {
// XXX: Add named region objects
// XXX: Until entry loading is complete just claim it is loaded
Entry->second->NamedJobRefCountMutex.unlock();
}
// XXX: Until entry loading is complete just claim it is loaded
Entry->second->NamedJobRefCountMutex.unlock();
}
void NamedRegionObjectHandler::RemoveNamedRegionObject(uintptr_t Base, uintptr_t Size, fextl::unique_ptr<CodeRegionEntry> Entry) {
// XXX: Remove named region objects
void NamedRegionObjectHandler::RemoveNamedRegionObject(uintptr_t Base, uintptr_t Size, fextl::unique_ptr<CodeRegionEntry> Entry) {
// XXX: Remove named region objects
// XXX: Until entry loading is complete just claim it is loaded
Entry->NamedJobRefCountMutex.unlock();
}
// XXX: Until entry loading is complete just claim it is loaded
Entry->NamedJobRefCountMutex.unlock();
}
void NamedRegionObjectHandler::HandleNamedRegionObjectJobs() {
// Walk through all of our jobs sequentially until the work queue is empty
while (NamedWorkQueueJobs.load()) {
fextl::unique_ptr<AsyncJobHandler::NamedRegionWorkItem> WorkItem;
void NamedRegionObjectHandler::HandleNamedRegionObjectJobs() {
// Walk through all of our jobs sequentially until the work queue is empty
while (NamedWorkQueueJobs.load()) {
fextl::unique_ptr<AsyncJobHandler::NamedRegionWorkItem> WorkItem;
{
// Lock the work queue mutex for a short moment and grab an item from the list
std::unique_lock lk {NamedWorkQueueMutex};
size_t WorkItems = WorkQueue.size();
if (WorkItems != 0) {
WorkItem = std::move(WorkQueue.front());
WorkQueue.pop();
}
// Atomically update the number of jobs
--NamedWorkQueueJobs;
{
// Lock the work queue mutex for a short moment and grab an item from the list
std::unique_lock lk {NamedWorkQueueMutex};
size_t WorkItems = WorkQueue.size();
if (WorkItems != 0) {
WorkItem = std::move(WorkQueue.front());
WorkQueue.pop();
}
if (WorkItem) {
if (WorkItem->GetType() == AsyncJobHandler::NamedRegionJobType::JOB_ADD_NAMED_REGION) {
auto WorkAdd = static_cast<AsyncJobHandler::WorkItemAddNamedRegion *>(WorkItem.get());
AddNamedRegionObject(WorkAdd->Entry, WorkAdd->BaseFilename, WorkAdd->Filename, WorkAdd->Executable);
}
// Atomically update the number of jobs
--NamedWorkQueueJobs;
}
if (WorkItem->GetType() == AsyncJobHandler::NamedRegionJobType::JOB_REMOVE_NAMED_REGION) {
auto WorkRemove = static_cast<AsyncJobHandler::WorkItemRemoveNamedRegion *>(WorkItem.get());
RemoveNamedRegionObject(WorkRemove->Base, WorkRemove->Size, std::move(WorkRemove->Entry));
}
if (WorkItem) {
if (WorkItem->GetType() == AsyncJobHandler::NamedRegionJobType::JOB_ADD_NAMED_REGION) {
auto WorkAdd = static_cast<AsyncJobHandler::WorkItemAddNamedRegion*>(WorkItem.get());
AddNamedRegionObject(WorkAdd->Entry, WorkAdd->BaseFilename, WorkAdd->Filename, WorkAdd->Executable);
}
if (WorkItem->GetType() == AsyncJobHandler::NamedRegionJobType::JOB_REMOVE_NAMED_REGION) {
auto WorkRemove = static_cast<AsyncJobHandler::WorkItemRemoveNamedRegion*>(WorkItem.get());
RemoveNamedRegionObject(WorkRemove->Base, WorkRemove->Size, std::move(WorkRemove->Entry));
}
}
}
}
} // namespace FEXCore::CodeSerialize
@@ -6,80 +6,80 @@
#include <FEXCore/Utils/Threads.h>
namespace {
static void* ThreadHandler(void *Arg) {
FEXCore::CodeSerialize::CodeObjectSerializeService *This = reinterpret_cast<FEXCore::CodeSerialize::CodeObjectSerializeService*>(Arg);
This->ExecutionThread();
return nullptr;
}
static void* ThreadHandler(void* Arg) {
FEXCore::CodeSerialize::CodeObjectSerializeService* This = reinterpret_cast<FEXCore::CodeSerialize::CodeObjectSerializeService*>(Arg);
This->ExecutionThread();
return nullptr;
}
} // namespace
namespace FEXCore::CodeSerialize {
CodeObjectSerializeService::CodeObjectSerializeService(FEXCore::Context::ContextImpl *ctx)
: CTX {ctx}
, AsyncHandler { &NamedRegionHandler , this }
, NamedRegionHandler { ctx } {
Initialize();
CodeObjectSerializeService::CodeObjectSerializeService(FEXCore::Context::ContextImpl* ctx)
: CTX {ctx}
, AsyncHandler {&NamedRegionHandler, this}
, NamedRegionHandler {ctx} {
Initialize();
}
void CodeObjectSerializeService::Shutdown() {
if (CTX->Config.CacheObjectCodeCompilation() == FEXCore::Config::ConfigObjectCodeHandler::CONFIG_NONE) {
return;
}
void CodeObjectSerializeService::Shutdown() {
if (CTX->Config.CacheObjectCodeCompilation() == FEXCore::Config::ConfigObjectCodeHandler::CONFIG_NONE) {
return;
}
WorkerThreadShuttingDown = true;
WorkerThreadShuttingDown = true;
// Kick the working thread
WorkAvailable.NotifyAll();
// Kick the working thread
WorkAvailable.NotifyAll();
if (WorkerThread->joinable()) {
// Wait for worker thread to close down
WorkerThread->join(nullptr);
}
}
void CodeObjectSerializeService::Initialize() {
// Add a canary so we don't crash on empty map iterator handling
auto it = AddressToEntryMap.insert_or_assign(~0ULL, fextl::make_unique<CodeRegionEntry>());
UnrelocatedAddressToEntryMap.insert_or_assign(~0ULL, it.first->second.get());
uint64_t OldMask = FEXCore::Threads::SetSignalMask(~0ULL);
WorkerThread = FEXCore::Threads::Thread::Create(ThreadHandler, this);
FEXCore::Threads::SetSignalMask(OldMask);
}
void CodeObjectSerializeService::DoCodeRegionClosure(uint64_t Base, CodeRegionEntry *it) {
if (Base == ~0ULL) {
// Don't do closure on canary
return;
}
// XXX: Do code region closure
}
CodeObjectFileSection const *CodeObjectSerializeService::FetchCodeObjectFromCache(uint64_t GuestRIP) {
// XXX: Actually fetch code objects from cache
return nullptr;
}
void CodeObjectSerializeService::ExecutionThread() {
// Set our thread name so we can see its relation
FEXCore::Threads::SetThreadName("ObjectCodeSeri\0");
while (WorkerThreadShuttingDown.load() != true) {
// Wait for work
WorkAvailable.Wait();
// Handle named region async jobs first. Highest priority
NamedRegionHandler.HandleNamedRegionObjectJobs();
// XXX: Handle code serialization jobs second.
}
// Do final code region closures on thread shutdown
for (auto &it : AddressToEntryMap) {
DoCodeRegionClosure(it.first, it.second.get());
}
// Safely clear our maps now
AddressToEntryMap.clear();
UnrelocatedAddressToEntryMap.clear();
if (WorkerThread->joinable()) {
// Wait for worker thread to close down
WorkerThread->join(nullptr);
}
}
void CodeObjectSerializeService::Initialize() {
// Add a canary so we don't crash on empty map iterator handling
auto it = AddressToEntryMap.insert_or_assign(~0ULL, fextl::make_unique<CodeRegionEntry>());
UnrelocatedAddressToEntryMap.insert_or_assign(~0ULL, it.first->second.get());
uint64_t OldMask = FEXCore::Threads::SetSignalMask(~0ULL);
WorkerThread = FEXCore::Threads::Thread::Create(ThreadHandler, this);
FEXCore::Threads::SetSignalMask(OldMask);
}
void CodeObjectSerializeService::DoCodeRegionClosure(uint64_t Base, CodeRegionEntry* it) {
if (Base == ~0ULL) {
// Don't do closure on canary
return;
}
// XXX: Do code region closure
}
const CodeObjectFileSection* CodeObjectSerializeService::FetchCodeObjectFromCache(uint64_t GuestRIP) {
// XXX: Actually fetch code objects from cache
return nullptr;
}
void CodeObjectSerializeService::ExecutionThread() {
// Set our thread name so we can see its relation
FEXCore::Threads::SetThreadName("ObjectCodeSeri\0");
while (WorkerThreadShuttingDown.load() != true) {
// Wait for work
WorkAvailable.Wait();
// Handle named region async jobs first. Highest priority
NamedRegionHandler.HandleNamedRegionObjectJobs();
// XXX: Handle code serialization jobs second.
}
// Do final code region closures on thread shutdown
for (auto& it : AddressToEntryMap) {
DoCodeRegionClosure(it.first, it.second.get());
}
// Safely clear our maps now
AddressToEntryMap.clear();
UnrelocatedAddressToEntryMap.clear();
}
} // namespace FEXCore::CodeSerialize
@@ -17,445 +17,441 @@
#include <shared_mutex>
namespace FEXCore::CodeSerialize {
// XXX: Does this need to be signal safe?
using CodeSerializationMutex = std::shared_mutex;
struct CodeSerializationData {
};
// XXX: Does this need to be signal safe?
using CodeSerializationMutex = std::shared_mutex;
struct CodeSerializationData {};
struct CodeObjectFileSection {
bool Serialized;
bool Invalid;
const CodeSerializationData *Data;
const char *HostCode;
uint64_t NumRelocations;
const char *Relocations;
};
struct CodeObjectFileSection {
bool Serialized;
bool Invalid;
const CodeSerializationData* Data;
const char* HostCode;
uint64_t NumRelocations;
const char* Relocations;
};
/**
* @brief This is the file header that lives at the start of an object cache file
*
* This header is updated from multiple processes!
* Care must be taken to use OS locks when updating the file backing including this header
*/
struct CodeObjectSerializationHeader {
// The configuration that this file has
CodeObjectSerializationConfig Config;
// The original RIP that this object section was mapped at
uint64_t OriginalBase {};
// The original offset in to the file that this object section was loaded from
uint64_t OriginalOffset {};
// Total amount of code that should be in this file
uint64_t TotalCodeSize {};
// Used to reserve the TSL map
uint64_t NumCodeEntries {};
// The number of relocations that point to this section
uint64_t NumRelocationsTo {};
// Total relocations in this file
uint64_t TotalRelocationsCount {};
};
struct CodeRegionEntry {
/**
* @name Threaded initialization objects for the initial object creation
* @{ */
// Base address in memory where the code region is at
uint64_t Base {};
// Size of this code entry
uint64_t Size {};
// The offset inside the file that is mapped to Base
uint64_t Offset {};
// Filename of the object
fextl::string Filename {};
CodeObjectSerializationHeader EntryHeader {};
/** @} */
// The filename of the object cache for this entry
fextl::string ObjectEntrySourceFilename {};
// In the case of file corruption that we can detect, we can disable serialization early for an entry
// We should be resiliant to corruption but things happen
bool StillSerializing {true};
// Long lived FD for serialization if we have multiple jobs to serialize
// Bursts of code entries are common and this reduces file lock overhead
//
// Especially useful over network mounts where file locks are very slow
int CurrentSerializedFD {-1};
/**
* @brief This is the file header that lives at the start of an object cache file
*
* This header is updated from multiple processes!
* Care must be taken to use OS locks when updating the file backing including this header
*/
struct CodeObjectSerializationHeader {
// The configuration that this file has
CodeObjectSerializationConfig Config;
// The original RIP that this object section was mapped at
uint64_t OriginalBase{};
// The original offset in to the file that this object section was loaded from
uint64_t OriginalOffset{};
// Total amount of code that should be in this file
uint64_t TotalCodeSize{};
// Used to reserve the TSL map
uint64_t NumCodeEntries{};
// The number of relocations that point to this section
uint64_t NumRelocationsTo{};
// Total relocations in this file
uint64_t TotalRelocationsCount{};
};
* @name Objects required to sync objects between threads
* @{ */
// Refcount for the number of outstanding code entries waiting to be written for this object section
CodeSerializationMutex ObjectJobRefCountMutex;
struct CodeRegionEntry {
/**
* @name Threaded initialization objects for the initial object creation
* @{ */
// Base address in memory where the code region is at
uint64_t Base{};
// Size of this code entry
uint64_t Size{};
// The offset inside the file that is mapped to Base
uint64_t Offset{};
// Filename of the object
fextl::string Filename{};
CodeObjectSerializationHeader EntryHeader{};
/** @} */
// The filename of the object cache for this entry
fextl::string ObjectEntrySourceFilename{};
// In the case of file corruption that we can detect, we can disable serialization early for an entry
// We should be resiliant to corruption but things happen
bool StillSerializing {true};
// Long lived FD for serialization if we have multiple jobs to serialize
// Bursts of code entries are common and this reduces file lock overhead
//
// Especially useful over network mounts where file locks are very slow
int CurrentSerializedFD {-1};
/**
* @name Objects required to sync objects between threads
* @{ */
// Refcount for the number of outstanding code entries waiting to be written for this object section
CodeSerializationMutex ObjectJobRefCountMutex;
// Refcount for outstanding named object region entry loading itself
// Will block JIT code cache look up when this has a unique_lock held
CodeSerializationMutex NamedJobRefCountMutex;
/** @} */
/**
* @name Object Entry data management
* @{ */
/**
* @name This is the raw file data that we loaded from the code region entry file
* @{ */
char *CodeData{};
size_t FileSize{};
fextl::vector<CodeObjectFileSection> FileCodeSections;
/** @} */
// This per section map takes the most time to load and needs to be quick
// This is the map of all code segments for this entry
fextl::robin_map<uint64_t, CodeObjectFileSection*> SectionLookupMap{};
/** @} */
// Default initialization
CodeRegionEntry() = default;
// Initializer specifically for threaded loading
CodeRegionEntry(uint64_t Base,
uint64_t Size,
uint64_t Offset,
fextl::string const &Filename,
CodeObjectSerializationHeader const &DefaultHeader)
: Base {Base}
, Size {Size}
, Offset {Offset}
, Filename {Filename}
, EntryHeader {DefaultHeader} {
}
};
// Map type must use an interator that isn't invalidation on erase/insert
using CodeRegionMapType = fextl::map<uint64_t, fextl::unique_ptr<CodeRegionEntry>>;
using CodeRegionPtrMapType = fextl::map<uint64_t, CodeRegionEntry*>;
class NamedRegionObjectHandler;
class CodeObjectSerializeService;
class AsyncJobHandler final {
public:
/**
* @brief Structure containing all the data required to async serialize code objects
*/
struct SerializationJobData {
uint64_t GuestRIP; ///< The RIP for the guest
// XXX: Support multiblock
uint64_t GuestCodeLength; ///< The Guest's code length
uint64_t GuestCodeHash; ///< Hash of the guest code
void *HostCodeBegin; ///< Host JIT code starting memory address
size_t HostCodeLength; ///< Host JIT code length
uint64_t HostCodeHash; ///< Host JIT code hash before any backpatching
// This is the thread specific ref counter for outstanding jobs.
// This shared mutex is incremented when the job is added, then decremented when the job is complete.
// If a thread is shutting down or clearing code cache then the thread will pull a unique lock on this mutex.
// This way it will wait until the async job handler is complete with it.
CodeSerializationMutex *ThreadJobRefCount;
// These are the reolocations for this serialization job
// Relatively small number of entries most of the time
fextl::vector<FEXCore::CPU::Relocation> Relocations;
/**
* @name Objects filled in from the Code Object Serialization service when a job is added
* @{ */
// This is the code region's ref counter for outstanding jobs.
// This shared mutex is incremented when the job is added, then decremented when the job is complete.
// If a named region is being removed then a unique lock will be pulled to wait for all jobs to complete and no new jobs to be added.
CodeSerializationMutex *ObjectJobRefCountMutexPtr;
// This is the code region iterator to reduce the number of map lookups
// This will remain valid while jobs are outstanding for this region
CodeRegionMapType::iterator CodeRegionIterator;
/** @} */
};
AsyncJobHandler(NamedRegionObjectHandler *NamedRegionHandler, CodeObjectSerializeService *CodeObjectCacheService)
: NamedRegionHandler {NamedRegionHandler}
, CodeObjectCacheService {CodeObjectCacheService} {}
protected:
friend class CodeObjectSerializeService;
friend class NamedRegionObjectHandler;
/**
* @name Async job submission functions
* @{ */
void AsyncAddNamedRegionJob(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const fextl::string &filename);
void AsyncRemoveNamedRegionJob(uintptr_t Base, uintptr_t Size);
void AsyncAddSerializationJob(fextl::unique_ptr<SerializationJobData> Data);
/** @} */
/**
* @name Async named region handling
* @{ */
/**
* @brief The async named region jobs to handle.
*
* Only two, Code serialization goes in to a different queue.
*/
enum class NamedRegionJobType {
JOB_ADD_NAMED_REGION,
JOB_REMOVE_NAMED_REGION,
};
class NamedRegionWorkItem {
public:
NamedRegionJobType GetType() const { return Type; }
protected:
friend class WorkItemAddNamedRegion;
NamedRegionWorkItem(NamedRegionJobType type)
: Type {type} {}
private:
NamedRegionJobType Type;
};
class WorkItemAddNamedRegion : public NamedRegionWorkItem {
public:
WorkItemAddNamedRegion(const fextl::string &base, const fextl::string &filename, bool executable, CodeRegionMapType::iterator entry)
: NamedRegionWorkItem {NamedRegionJobType::JOB_ADD_NAMED_REGION}
, BaseFilename {base}
, Filename {filename}
, Executable {executable}
, Entry {entry}
{}
const fextl::string BaseFilename;
const fextl::string Filename;
bool Executable;
CodeRegionMapType::iterator Entry;
};
class WorkItemRemoveNamedRegion : public NamedRegionWorkItem {
public:
WorkItemRemoveNamedRegion(uint64_t base, uint64_t size, fextl::unique_ptr<CodeRegionEntry> entry)
: NamedRegionWorkItem {NamedRegionJobType::JOB_REMOVE_NAMED_REGION}
, Base {base}
, Size {size}
, Entry {std::move(entry)} {}
uint64_t Base;
uint64_t Size;
fextl::unique_ptr<CodeRegionEntry> Entry;
};
/** @} */
private:
NamedRegionObjectHandler *NamedRegionHandler;
CodeObjectSerializeService *CodeObjectCacheService;
};
class NamedRegionObjectHandler final {
public:
NamedRegionObjectHandler(FEXCore::Context::ContextImpl *ctx);
void HandleNamedRegionObjectJobs();
CodeObjectSerializationConfig const &GetDefaultSerializationConfig() const {
return DefaultSerializationConfig;
}
protected:
friend class AsyncJobHandler;
// Return a default code header based off the default serialization config
CodeObjectSerializationHeader DefaultCodeHeader(uint64_t Base, uint64_t Offset) const {
return CodeObjectSerializationHeader {
.Config = DefaultSerializationConfig,
.OriginalBase = Base,
.OriginalOffset = Offset,
.NumCodeEntries = 0,
.NumRelocationsTo = 0,
.TotalRelocationsCount = 0,
};
}
/**
* @brief Adds an asynchronous add named region work item to the object queue
*
* This adds the job that will do the loading of file resources and data tracking.
*/
void AsyncAddNamedRegionWorkItem(const fextl::string &base, const fextl::string &filename, bool executable, CodeRegionMapType::iterator entry) {
std::unique_lock lk {NamedWorkQueueMutex};
WorkQueue.emplace(fextl::make_unique<AsyncJobHandler::WorkItemAddNamedRegion> (
base,
filename,
executable,
entry
));
++NamedWorkQueueJobs;
}
void AsyncRemoveNamedRegionWorkItem(uint64_t Base, uint64_t Size, fextl::unique_ptr<CodeRegionEntry> Entry) {
std::unique_lock lk {NamedWorkQueueMutex};
WorkQueue.emplace(fextl::make_unique<AsyncJobHandler::WorkItemRemoveNamedRegion> (
Base,
Size,
std::move(Entry)
));
++NamedWorkQueueJobs;
}
private:
// Code version. If the code emission changes then this needs to increment
constexpr static uint32_t CODE_VERSION = 0x0;
// Default cookie header for the file header
constexpr static uint64_t CODE_COOKIE = FEXCore::IR::COOKIE_VERSION("FEXC", CODE_VERSION);
// Code serialization config for our current process configuration
CodeObjectSerializationConfig DefaultSerializationConfig;
// Atomic counter for number of jobs in the queue without needing to pull the mutex to check
std::atomic<uint64_t> NamedWorkQueueJobs{};
// Mutex for ading new jobs to the work queue
std::mutex NamedWorkQueueMutex{};
// The job queue itself
// Jobs get consumed as a FIFO
// Jobs always get appended to the end
fextl::queue<fextl::unique_ptr<AsyncJobHandler::NamedRegionWorkItem>> WorkQueue{};
/**
* @name Named Region object handling
* @{ */
void AddNamedRegionObject(CodeRegionMapType::iterator Entry, const fextl::string &base_filename, const fextl::string &filename, bool Executable);
void RemoveNamedRegionObject(uintptr_t Base, uintptr_t Size, fextl::unique_ptr<CodeRegionEntry> Entry);
/** @} */
};
// Refcount for outstanding named object region entry loading itself
// Will block JIT code cache look up when this has a unique_lock held
CodeSerializationMutex NamedJobRefCountMutex;
/** @} */
/**
* @brief Context specific code object serialization class
*
* Contains everything required for FEXCore to serialize code objects
* @name Object Entry data management
* @{ */
/**
* @name This is the raw file data that we loaded from the code region entry file
* @{ */
char* CodeData {};
size_t FileSize {};
fextl::vector<CodeObjectFileSection> FileCodeSections;
/** @} */
// This per section map takes the most time to load and needs to be quick
// This is the map of all code segments for this entry
fextl::robin_map<uint64_t, CodeObjectFileSection*> SectionLookupMap {};
/** @} */
// Default initialization
CodeRegionEntry() = default;
// Initializer specifically for threaded loading
CodeRegionEntry(uint64_t Base, uint64_t Size, uint64_t Offset, const fextl::string& Filename, const CodeObjectSerializationHeader& DefaultHeader)
: Base {Base}
, Size {Size}
, Offset {Offset}
, Filename {Filename}
, EntryHeader {DefaultHeader} {}
};
// Map type must use an interator that isn't invalidation on erase/insert
using CodeRegionMapType = fextl::map<uint64_t, fextl::unique_ptr<CodeRegionEntry>>;
using CodeRegionPtrMapType = fextl::map<uint64_t, CodeRegionEntry*>;
class NamedRegionObjectHandler;
class CodeObjectSerializeService;
class AsyncJobHandler final {
public:
/**
* @brief Structure containing all the data required to async serialize code objects
*/
class CodeObjectSerializeService final {
public:
CodeObjectSerializeService(FEXCore::Context::ContextImpl *ctx);
struct SerializationJobData {
uint64_t GuestRIP; ///< The RIP for the guest
// XXX: Support multiblock
uint64_t GuestCodeLength; ///< The Guest's code length
uint64_t GuestCodeHash; ///< Hash of the guest code
/**
* @brief Initialize the internal interface
*
* Is a public interface to allow the service to reinitialize after forking
*/
void Initialize();
void* HostCodeBegin; ///< Host JIT code starting memory address
size_t HostCodeLength; ///< Host JIT code length
uint64_t HostCodeHash; ///< Host JIT code hash before any backpatching
/**
* @brief Safely shut down the Code Object serialization service.
*
* This service needs to be resiliant to application crashes, but shutting down safely is still preferred.
*/
void Shutdown();
// This is the thread specific ref counter for outstanding jobs.
// This shared mutex is incremented when the job is added, then decremented when the job is complete.
// If a thread is shutting down or clearing code cache then the thread will pull a unique lock on this mutex.
// This way it will wait until the async job handler is complete with it.
CodeSerializationMutex* ThreadJobRefCount;
/**
* @name Async interface
* @{ */
/**
* @brief Loads a named region in to the code serialization service. As async as possible.
*
* @param Base - Virtual address that this named region is loaded
* @param Size - The size of the region
* @param Offset - The offset from the file
* @param filename - The filename itself
*/
void AsyncAddNamedRegionJob(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const fextl::string &filename) {
AsyncHandler.AsyncAddNamedRegionJob(Base, Size, Offset, filename);
}
// These are the reolocations for this serialization job
// Relatively small number of entries most of the time
fextl::vector<FEXCore::CPU::Relocation> Relocations;
/**
* @brief Unloads a named region from the code serialization service. As async as possible.
*
* @param Base - Virtual address of the named region
* @param Size - The size of the region
*/
void AsyncRemoveNamedRegionJob(uintptr_t Base, uintptr_t Size) {
AsyncHandler.AsyncRemoveNamedRegionJob(Base, Size);
}
/**
* @name Objects filled in from the Code Object Serialization service when a job is added
* @{ */
// This is the code region's ref counter for outstanding jobs.
// This shared mutex is incremented when the job is added, then decremented when the job is complete.
// If a named region is being removed then a unique lock will be pulled to wait for all jobs to complete and no new jobs to be added.
CodeSerializationMutex* ObjectJobRefCountMutexPtr;
/**
* @brief Adds a code object serialization job. As async as possible.
* Code hashing happens prior to async job serialization to catch invalidations due to backpatching.
*
* @param Data - A fully filled out struct containing all the code serialization
*/
void AsyncAddSerializationJob(fextl::unique_ptr<AsyncJobHandler::SerializationJobData> Data) {
AsyncHandler.AsyncAddSerializationJob(std::move(Data));
}
/** @} */
/**
* @name Synchronous interface
* @{ */
/**
* @brief Synchronously waits for this thread's job queue to become empty.
*
* This is necessary for when a thread is shutting down
*
* @param ThreadJobRefCount - The shared mutex to wait on until to be empty
*/
static void WaitForEmptyJobQueue(CodeSerializationMutex *ThreadJobRefCount) {
// Once the shared mutex is empty this unique lock will be gained
std::unique_lock lk {*ThreadJobRefCount};
}
/**
* @brief Fetches object code from the Code Object Cache for JIT.
*
* @param GuestRIP - Which GuestRIP to search the cache for
*
* @return Data required for the JIT to relocate the Object code.
*/
CodeObjectFileSection const *FetchCodeObjectFromCache(uint64_t GuestRIP);
/** @} */
// Public for threading
void ExecutionThread();
protected:
friend class AsyncJobHandler;
/**
* @brief Safely closes out code object regions from the map
*
* @param it - iterator to do a closure on
*/
void DoCodeRegionClosure(uint64_t Base, CodeRegionEntry *it);
CodeSerializationMutex &GetEntryMapMutex() { return EntryMapMutex; }
CodeSerializationMutex &GetUnrelocatedEntryMapMutex() { return EntryMapMutex; }
CodeRegionMapType &GetEntryMap() { return AddressToEntryMap; }
CodeRegionPtrMapType &GetUnrelocatedEntryMap() { return UnrelocatedAddressToEntryMap; }
/**
* @brief Notify the async thread that it has work to do
*/
void NotifyWork() { WorkAvailable.NotifyOne(); }
private:
FEXCore::Context::ContextImpl *CTX;
Event WorkAvailable{};
fextl::unique_ptr<FEXCore::Threads::Thread> WorkerThread;
std::atomic_bool WorkerThreadShuttingDown {false};
AsyncJobHandler AsyncHandler;
NamedRegionObjectHandler NamedRegionHandler;
// Mutex to hold when modifying the entry maps
CodeSerializationMutex EntryMapMutex;
CodeSerializationMutex UnrelocatedEntryMapMutex;
// Entry maps
CodeRegionMapType AddressToEntryMap;
CodeRegionPtrMapType UnrelocatedAddressToEntryMap;
// This is the code region iterator to reduce the number of map lookups
// This will remain valid while jobs are outstanding for this region
CodeRegionMapType::iterator CodeRegionIterator;
/** @} */
};
}
AsyncJobHandler(NamedRegionObjectHandler* NamedRegionHandler, CodeObjectSerializeService* CodeObjectCacheService)
: NamedRegionHandler {NamedRegionHandler}
, CodeObjectCacheService {CodeObjectCacheService} {}
protected:
friend class CodeObjectSerializeService;
friend class NamedRegionObjectHandler;
/**
* @name Async job submission functions
* @{ */
void AsyncAddNamedRegionJob(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const fextl::string& filename);
void AsyncRemoveNamedRegionJob(uintptr_t Base, uintptr_t Size);
void AsyncAddSerializationJob(fextl::unique_ptr<SerializationJobData> Data);
/** @} */
/**
* @name Async named region handling
* @{ */
/**
* @brief The async named region jobs to handle.
*
* Only two, Code serialization goes in to a different queue.
*/
enum class NamedRegionJobType {
JOB_ADD_NAMED_REGION,
JOB_REMOVE_NAMED_REGION,
};
class NamedRegionWorkItem {
public:
NamedRegionJobType GetType() const {
return Type;
}
protected:
friend class WorkItemAddNamedRegion;
NamedRegionWorkItem(NamedRegionJobType type)
: Type {type} {}
private:
NamedRegionJobType Type;
};
class WorkItemAddNamedRegion : public NamedRegionWorkItem {
public:
WorkItemAddNamedRegion(const fextl::string& base, const fextl::string& filename, bool executable, CodeRegionMapType::iterator entry)
: NamedRegionWorkItem {NamedRegionJobType::JOB_ADD_NAMED_REGION}
, BaseFilename {base}
, Filename {filename}
, Executable {executable}
, Entry {entry} {}
const fextl::string BaseFilename;
const fextl::string Filename;
bool Executable;
CodeRegionMapType::iterator Entry;
};
class WorkItemRemoveNamedRegion : public NamedRegionWorkItem {
public:
WorkItemRemoveNamedRegion(uint64_t base, uint64_t size, fextl::unique_ptr<CodeRegionEntry> entry)
: NamedRegionWorkItem {NamedRegionJobType::JOB_REMOVE_NAMED_REGION}
, Base {base}
, Size {size}
, Entry {std::move(entry)} {}
uint64_t Base;
uint64_t Size;
fextl::unique_ptr<CodeRegionEntry> Entry;
};
/** @} */
private:
NamedRegionObjectHandler* NamedRegionHandler;
CodeObjectSerializeService* CodeObjectCacheService;
};
class NamedRegionObjectHandler final {
public:
NamedRegionObjectHandler(FEXCore::Context::ContextImpl* ctx);
void HandleNamedRegionObjectJobs();
const CodeObjectSerializationConfig& GetDefaultSerializationConfig() const {
return DefaultSerializationConfig;
}
protected:
friend class AsyncJobHandler;
// Return a default code header based off the default serialization config
CodeObjectSerializationHeader DefaultCodeHeader(uint64_t Base, uint64_t Offset) const {
return CodeObjectSerializationHeader {
.Config = DefaultSerializationConfig,
.OriginalBase = Base,
.OriginalOffset = Offset,
.NumCodeEntries = 0,
.NumRelocationsTo = 0,
.TotalRelocationsCount = 0,
};
}
/**
* @brief Adds an asynchronous add named region work item to the object queue
*
* This adds the job that will do the loading of file resources and data tracking.
*/
void AsyncAddNamedRegionWorkItem(const fextl::string& base, const fextl::string& filename, bool executable, CodeRegionMapType::iterator entry) {
std::unique_lock lk {NamedWorkQueueMutex};
WorkQueue.emplace(fextl::make_unique<AsyncJobHandler::WorkItemAddNamedRegion>(base, filename, executable, entry));
++NamedWorkQueueJobs;
}
void AsyncRemoveNamedRegionWorkItem(uint64_t Base, uint64_t Size, fextl::unique_ptr<CodeRegionEntry> Entry) {
std::unique_lock lk {NamedWorkQueueMutex};
WorkQueue.emplace(fextl::make_unique<AsyncJobHandler::WorkItemRemoveNamedRegion>(Base, Size, std::move(Entry)));
++NamedWorkQueueJobs;
}
private:
// Code version. If the code emission changes then this needs to increment
constexpr static uint32_t CODE_VERSION = 0x0;
// Default cookie header for the file header
constexpr static uint64_t CODE_COOKIE = FEXCore::IR::COOKIE_VERSION("FEXC", CODE_VERSION);
// Code serialization config for our current process configuration
CodeObjectSerializationConfig DefaultSerializationConfig;
// Atomic counter for number of jobs in the queue without needing to pull the mutex to check
std::atomic<uint64_t> NamedWorkQueueJobs {};
// Mutex for ading new jobs to the work queue
std::mutex NamedWorkQueueMutex {};
// The job queue itself
// Jobs get consumed as a FIFO
// Jobs always get appended to the end
fextl::queue<fextl::unique_ptr<AsyncJobHandler::NamedRegionWorkItem>> WorkQueue {};
/**
* @name Named Region object handling
* @{ */
void AddNamedRegionObject(CodeRegionMapType::iterator Entry, const fextl::string& base_filename, const fextl::string& filename, bool Executable);
void RemoveNamedRegionObject(uintptr_t Base, uintptr_t Size, fextl::unique_ptr<CodeRegionEntry> Entry);
/** @} */
};
/**
* @brief Context specific code object serialization class
*
* Contains everything required for FEXCore to serialize code objects
*/
class CodeObjectSerializeService final {
public:
CodeObjectSerializeService(FEXCore::Context::ContextImpl* ctx);
/**
* @brief Initialize the internal interface
*
* Is a public interface to allow the service to reinitialize after forking
*/
void Initialize();
/**
* @brief Safely shut down the Code Object serialization service.
*
* This service needs to be resiliant to application crashes, but shutting down safely is still preferred.
*/
void Shutdown();
/**
* @name Async interface
* @{ */
/**
* @brief Loads a named region in to the code serialization service. As async as possible.
*
* @param Base - Virtual address that this named region is loaded
* @param Size - The size of the region
* @param Offset - The offset from the file
* @param filename - The filename itself
*/
void AsyncAddNamedRegionJob(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const fextl::string& filename) {
AsyncHandler.AsyncAddNamedRegionJob(Base, Size, Offset, filename);
}
/**
* @brief Unloads a named region from the code serialization service. As async as possible.
*
* @param Base - Virtual address of the named region
* @param Size - The size of the region
*/
void AsyncRemoveNamedRegionJob(uintptr_t Base, uintptr_t Size) {
AsyncHandler.AsyncRemoveNamedRegionJob(Base, Size);
}
/**
* @brief Adds a code object serialization job. As async as possible.
* Code hashing happens prior to async job serialization to catch invalidations due to backpatching.
*
* @param Data - A fully filled out struct containing all the code serialization
*/
void AsyncAddSerializationJob(fextl::unique_ptr<AsyncJobHandler::SerializationJobData> Data) {
AsyncHandler.AsyncAddSerializationJob(std::move(Data));
}
/** @} */
/**
* @name Synchronous interface
* @{ */
/**
* @brief Synchronously waits for this thread's job queue to become empty.
*
* This is necessary for when a thread is shutting down
*
* @param ThreadJobRefCount - The shared mutex to wait on until to be empty
*/
static void WaitForEmptyJobQueue(CodeSerializationMutex* ThreadJobRefCount) {
// Once the shared mutex is empty this unique lock will be gained
std::unique_lock lk {*ThreadJobRefCount};
}
/**
* @brief Fetches object code from the Code Object Cache for JIT.
*
* @param GuestRIP - Which GuestRIP to search the cache for
*
* @return Data required for the JIT to relocate the Object code.
*/
const CodeObjectFileSection* FetchCodeObjectFromCache(uint64_t GuestRIP);
/** @} */
// Public for threading
void ExecutionThread();
protected:
friend class AsyncJobHandler;
/**
* @brief Safely closes out code object regions from the map
*
* @param it - iterator to do a closure on
*/
void DoCodeRegionClosure(uint64_t Base, CodeRegionEntry* it);
CodeSerializationMutex& GetEntryMapMutex() {
return EntryMapMutex;
}
CodeSerializationMutex& GetUnrelocatedEntryMapMutex() {
return EntryMapMutex;
}
CodeRegionMapType& GetEntryMap() {
return AddressToEntryMap;
}
CodeRegionPtrMapType& GetUnrelocatedEntryMap() {
return UnrelocatedAddressToEntryMap;
}
/**
* @brief Notify the async thread that it has work to do
*/
void NotifyWork() {
WorkAvailable.NotifyOne();
}
private:
FEXCore::Context::ContextImpl* CTX;
Event WorkAvailable {};
fextl::unique_ptr<FEXCore::Threads::Thread> WorkerThread;
std::atomic_bool WorkerThreadShuttingDown {false};
AsyncJobHandler AsyncHandler;
NamedRegionObjectHandler NamedRegionHandler;
// Mutex to hold when modifying the entry maps
CodeSerializationMutex EntryMapMutex;
CodeSerializationMutex UnrelocatedEntryMapMutex;
// Entry maps
CodeRegionMapType AddressToEntryMap;
CodeRegionPtrMapType UnrelocatedAddressToEntryMap;
};
} // namespace FEXCore::CodeSerialize
Loaded 100 of 1005 files, more files were not shown because too many files have changed in this diff. Show more