Switches to the new CDN which is significantly faster and has other
benefits.
In order to make sure we don't break old clients, switch to the new link
for a few months while leaving the old one operational.
The links file in the old CDN still points to the new rootfs links so
they get the performance improvement on old clients still.
Need to ensure that dirfd is AT_FDCWD and also need to check flags
correctly.
Flags were incorrectly checking mode for O_WRONLY and also we should
check for O_APPEND. Split it out to a helper function just so it is
easier to see what is going on.
Fixes the issue of proton not finding `/lib64/ld-linux-x86-64.so.2`
x86's REP STOS instruction is a memset (with element size!) with the
ability to choose a direction of execution.
Additionally it has a feature where if it faults part-way through the
copy, an application can catch the fault and continue afterwards to know
how many bytes got copied.
RCX is the counter which decrements for each element, and RDI is the
memory pointer. On fault these will reflect the last location that was
attempted to be written. FEX doesn't support this behaviour which makes
our lives easier.
Without supporting that feature, this turns in to a directional memset
by element size. Let's remove all the multiple blocks and just emit a
single IR operation to improve performance of the JIT.
Our generated code here was terrible, the IR was terrible, multiblock is
always slow with RA. Just a general overall improvement.
With profiling pressure-vessel this change deletes the hottest block that
appeared in the trace. This instruction is very commonly used for
memsetting a region to zero so it should be quite fast.
We can also optimize REP MOVS in the future with a Memcpy IR operation
in a similar fashion.
Additionally in the future these can be optimized to use ARM's new MOPS
instructions since the most common case is memset by byte. Which is when
we should expose the "Fast REP STOS" CPUID bit. Both setp/setm/sete and
cpyfp/cpyfm/cpyfe match `REP STOS` and `REP MOVS` respectively.
This operation directly matches what the x86 STOS instruction does
without supporting its faulting behaviour.
STOS faulting behaviour is that RCX and RDI get updated to the last word
written. Which is something that FEX hasn't ever supported.
In the case that a file is getting opened to be created or writable then
skip EmuFD and rootfs searching for this file.
This fixes an edge case where if FEX was run with an unpacked rootfs
that was writable then pressure-vessel would break.
Fixes pressure-vessel with unpacked rootfs.
Like with the vector registers, we can remove all implicit conversion
operators except the ones that convert down to the base PRegister class.
With this, all of the registers are now adequately constrained, so we
shouldn't have any wonky implicit conversions happening anymore.
Ensures that we're always explicit about the size of a register when
using APIs that enforce it.
The only implicit conversions we keep are conversions that convert down
to Register, but not anything that converts up the hierarchy or across
it.
A few vector instructions that take register lists often require
vector registers within the list to be sequential in the form of an
increasing list modulo the register file size.
For example:
v1, v2, v3, v4
v31, v0, v1, v2
both fit these requirements.
This will be used to enforce this restriction within the asserts from a
single place.
A large amount of size handling in this category is just decrementing
the size by 1, so we can tidy up a bunch of conditionals by just doing
that instead.
When FEXServer is daemonizing through an instance of FEXLoader or
FEXInterpreter, it would leave a zombie process which was waiting for us
to read the process status.
Since we don't care about the child status and don't want to get blocked
by waitpid, just ignore the signal.
This tells the kernel that we don't care about the signal and will kill
the zombie process immediately.
Didn't notice this before since FEXServer started failing to daemonize.
It looks like `SYSTEMD_EXEC_PID` can leak through to the executable
environment in regular situations. Instead let's key off of
`INVOCATATION_ID` which doesn't leak through.
Fixes an edge case behaviour where FEXServer wouldn't daemonize in some
systemd environments.
Fixes#2443
I found out with some profiling that this we were spending a decent
amount of time with the `openat` syscall in heavily utilized situations.
While not super common in active gameplay situations, it matters
significantly in loading screens that this is fairly optimal.
The bulk of the time is spent in the emulated files handler to ensure
that whatever path we are given, we can capture file paths that we need
to emulate. The largest contributor being the std::filesystem::canonical
function call.
A couple of optimizations in place here.
1) Do a quick hashmap check right at the start to see if we exactly fit
2) Change from `std::fs::canonical` to `realpath`
3) Switch `GetEmulatedFDPath` to not use optional so it stops building
on the stack
I'm still not super happy with the performance of `realpath` and also
not happy that we still need to use `lexically_normal` in one code path.
But short of writing a super hand-optimized `realpath` that fits our
constraints, I don't think we can do better.
Micro benchmark needs to test four different situations due to this
optimization.
1) Non-EmuFD path
2) Non-EmuFD path with dirfs
3) EmuFD path
4) EmuFD path with dirfs
And the performance improvement for each situation respectively
1) 12% performance improvement
- 213413 openat syscalls/s -> 238999 syscalls/s
2) 17% performance improvement
- 202085 openat syscalls/s -> 237309 syscalls/s
3) 17% performance improvement (/proc/cpuinfo)
- 56616 openat syscalls/s -> 66231 syscalls/s
- Includes overhead of generating temp FD and close syscall
4) 5% performance improvement (/proc/cpuinfo)
- 51080 openat syscalls/s -> 53956 syscalls/s
- Includes overhead of generating temp FD and close syscall
And for sake of comparison to the non-emulated system; My test system
can hit around 1-1.1 million openat syscalls per second in the same
microbench.
Nice little performance uplift.
This is known to cause issues in some cases. We hit the first game that
checks for this bit and early exits if it is found.
Lets the MMORPG Tibia run in non-VM situations.
Looks like they have more checks for VMs other than hypervisor bit, so
running under Parallels still won't work. Running on bare Linux is fine.
Centralizes all the shifting behavior and whatnot into a single
function, making everything much more localized.
Also gets rid of a lot of magic constants related to the encoding limits
of immediates.
When we were taking a signal that had a misaligned stack, we would store
the host stack at a weird offset.
After that point when we were trying to sigreturn we wouldn't know the
alignment of the stack coming back and we would try loading the host
stack from the wrong offset. Easy fix is to just align the host stack
location.
Fixes Ender Lilies, which was consistently crashing from a SIGCHLD due
to having a misaligned stack.
Side-change: Move the cookie check to the start of the restore. Doesn't
make sense to check the cookie after restoring state since it could be
quite wrong.
Now that we've moved MOVHPS over to sharing the implementation of
MOVHPD, the FLAGS_SF_HIGH_XMM_REG is now unused.
Since we're supporting AVX, this flag is kind of weird in terms of
behavior, since what determines the high part of a register is now
situationally different.
Also it's much more explicit to perform the insert directly in the
implementation of instructions, than relying on a flag to do it for us.
So, instead of keeping it around, we can reclaim it as unused for use
with any necessary behavior that we would require in the future.
These instructions essentially have the same behavior. This also allows
us to remove the only used instance of FLAGS_SF_HIGH_XMM_REG, which,
given that we now support AVX, has ambiguous use.
While we're at it, we can expand the tests to make use of the store to
memory variant.
Also removes an erroneous copy-pasted comment about ZEXTing. This is
from the MOVQ implementation function. MOVHPS/MOVHPD don't do any
ZEXTing, they either store to memory or insert into a register.
Will catch cases where registers aren't consecutive in the constructive
variant. While we're at it, we can also amend EXT's similar but slightly wrong
consecutive check.
Also adds tests to ensure these corner-cases hold.
Performance stats up front:
This improves pressure-vessel startup time on my test device by 10.1%
Improving the startup time from 9.71425 seconds to 8.7421 seconds.
Most filesystem based syscalls support a file descriptor version with an
*at suffix. This allows us to do these syscalls with pathnames that are
relative to the directory FD that is passed to the syscall.
This is pretty much exactly what we want when we are searching for files
inside of our rootfs. The only quirk ends up being that we are getting
passed absolute paths. This ends up being very simple to workaround by
stripping off the front '/' character. Doing this is just offsetting the
pointer passed to the syscall by one byte.
This does require having two temporary buffers of size PATH_MAX passed
to the handler since just like in the other implementation, we need to
keep the previous result around. The difference being now that we aren't
doing a bunch of std::string temporary manipulation and now we are
returning one of the passed in buffers back depending on the result.
Moves some instruction helpers that existed outside of the private
section of the class back into them, so that we're not exposing
unnecessary things in the interface.
When executing a non-inlined syscall, we spill all static registers.
We weren't storing in to the thread context that we have done this.
If a signal occured between FEX returning from the syscall (after the
blr) and before the `FillStaticRegs` then the signal handler would get
the incorrect register state.
This typically manifested as Steam getting a SIGCHLD, trying to recover
the guest stack pointer, and it that pointer would be zero or some other
corrupt value. Thus crashing inside of the signal handler.
Surprising that we hadn't hit this way more before this point, must have
needed hardware that tickled the race condition *just* right.
This has been a long time coming. The C interface has been a thorn in
our side for no reason for a long time.
The purpose of this step is to remove the C interface without changing
behaviour as much as possible. This means that with this commit there
are still some bad practices but the remaining issues will be solved
with followup PRs.
Primarily, we still have a `DestroyContext(CTX)` static function which calls
the Context implementation's `DestroyContext` and does a raw C++ delete.
Follow up PR will remove that, but I didn't want to touch it yet since
it'll require checking to ensure the unique_ptr changes play nice with
our allocator hooking. Which this is already a huge PR without trying to
change behaviour.
Cortex-X1C and A78C are relatively minor changes to their non-C
counterparts. Support classifying them in case clang understands them.
Fixes a minor perf regression noticed on the Lenovo X13s while testing.
Take a leaf from the Vector ops and have the jump entry choose the IR
op.
Also generate one atomic op and modify the IR type in the locked memory
type just like the non locked memory path.
This class of instructions in the number one instruction type percentage
wise, so making this more optimal will be a win.
It's a fairly minor optimization so it should be a small impact.
This allows us to not update RIP on block entry, but still allow
reconstructing the RIP up until that point.
While still not full RIP reconstruction, this lets us update the signal
context's RIP just like the `x86dec_SynchronizeRIPOnAllBlocks` without
eating the cost of writing to RIP on block entry.
We can move the initial move down by SPLICE, which not only lets us turn
it into a MOVPRFX, but also we can safely move into the final
destination register directly, since we can be sure there's no
potential dependencies at this point
Fixes#2410
We were pulling the guest RSP before spilling static registers back to
the state.
Move this to after we spill SRA state to fix this bug.
Thanks to @ifquant for diving in, identifying, and finding the exact bug.
Because the binaries have no metadata, we allow a .json file to be
placed alongside a test indicating required features in a requirements
directory
We also check if the system itself supports those features and run tests
based off of that.
This will allow us to support operating on 256-bit vectors.
Currently only sets up the bits and pieces on the x86-64 side, since
facilities for testing the 256-bit operations on ARM isn't set up yet.
With this in place, a _VMov isn't necessary for variable blends anymore,
since the vector upper lanes are guaranteed to be zeroed out in the 128-bit case.
Currently unused (The full data gets thrown away after CompileCode is
called), but allows us to separate code and data in what `CompileCode`
returns.
This will allow us put a header on JIT blocks which will fix a long
outstanding bug where RIP isn't always synchronized on block entry, but
since it only needs to synchronize on signal we can rebuild in the
handler. This future task will remove the `86dec_SynchronizeRIPOnAllBlocks`
config option, but the data will also end up being used for more things
in the future.
SelectCC shift wasn't necessary since we just need to ensure the final
result is zero when or'd together.
Also operations calculating SF can just use a BFE instead of a shifts
with a constant. BFE by immediate is more efficiently encoded in our IR.
When used in conjuction with #2345 this is a useful way to enable
libSegFault in applications using application profiles.
Very useful for applications and games that use launcher scripts that
set LD_PRELOAD to nothing prior to launch.
A user was wanting this.
For the case that the 32-bit VDSO thunk library isn't available, have a
fallback that can work as well.
Otherwise 32-bit applications will just straight up crash on signal
return.
VDSO behaves like a raw syscall which doesn't set errno.
posix tests are testing that errno is set correctly.
Our VDSO handlers weren't wired up to return errors from VDSO correctly.
To handle this we need to have different handlers depending on if the
syscall being used comes from glibc or true VDSO.
This wasn't being uncovered previously since CI wasn't running with VDSO
thunks enabled, but now that it is this needs to be handled or CI will
fail.
This is required for backtrace to work correctly.
If we are using our custom instruction for returning from a signal, then
backtrace tries to read PC for the sigreturn code and finds our code,
breaking it.
Instead we now /correctly/ support using rt_sigreturn/sigreturn and the
restorer provided from the user.
To facilitate this, we now store a single 64-bit value on the stack to
return our host stack pointer to the correct location from before the
signal.
With cookie checking in place, we can know if an application betrays our
expectations and tries to pass its own signal frames.
If an application in the future /does/ try to pass its own signal
frames, that's unsafe and we cna deal with it then.
This fixes a bug where we are falling back down the default signal
delegator after a fatal error.
We need to reraise the event in the case that it didn't come from the
kernel.
Fixes backtrace crashing with incorrect signal when it tries to reraise
the signal that it handled using tgkill.
These need to be bit-exact following exactly what is shown in the
assembly.
libunwind parses where EIP is to see if it is in a stack frame.
Also needsto live in VDSO otherwise backtrace doesn't work.
The instructions allow specifying 64-bit element sizes.
With this, we can also completely remove the size checking from the
functions, since the general SVE integer reduction operation already
checks for invalid sizes for us.
Fixes this file compiling on musl at least.
POLLREMOVE usage here is technically incorrect as it shouldn't be OR'd
with other flags.
But it is also additionally wrong here because the Linux kernel doesn't
even support this flag anymore, so it doesn't change behaviour.
Fixes#2356
In the case of the `-y` option being used, it will auto say "yes", but
when presented with the distro list this doesn't work. This happens when
used on a distro that doesn't have an exact match to what we provide.
Exposes a new option that when presented the distro list, auto select
the first option. Solving this issue when automating.
I made the assumption from some bad historical knowledge that the kernel
will canonicalize relative filenames and symlinks for applications that
execute through execve.
This turns out to not be true. In fact it passes pathname untouched to
the interpreter. So we need to do an additional fix up on relative paths
to ensure glibc doesn't break.
Fixes a major bug that breaks a bunch of games.
1) The host library needs to be loaded in the global namespace.
2) We need to use `RTLD_DEFAULT` instead of querying the object
directly.
We need to load the host library in the global namespace so the symbols
end up in the global symbol table. This follows how all these symbols
/usually/ get loaded. Either by linking directly to the library or how
loaders will end up loading these.
We need to use RTLD_DEFAULT to follow symbol overriding rules that tend
to occur. For example, MangoHUD will LD_PRELOAD a library that provides
GLX and EGL symbols. Which FEX's thunk libraries need to pick up this
override.
If we are querying the host library directly then we fail to pickup
these overrides, thus breaking MangoHUD and other overlays.
Makes the behavior consistent with the x86 JIT.
We need to treat values larger than 31 as if they were 31 bit shifts in
order to handle sign-extending behavior properly.
Follow-up to #2327.
Split off from #2176 and improved.
32-bit signals are a bit more complex than 64-bit due to behaviour
changing depending on if `rt_sigaction` and `sigaction` syscall is used
and if `SA_SIGINFO` is passed in to the flags.
With `SA_SIGINFO` used, both turn in to an `RT` frame, which is encoded
differently than without `SA_SIGINFO`.
Additionally 32-bit signals support both regular Linux stack ABI and
`regparm(3)` ABI.
Without `SA_SIGINFO` then `siginfo_t` is removed from the signal handler
arguments, but most of the rest still remains.
Also two of the arguments to the signal handler are forced to be nullptr
with `regparm(3)`.
Split off from #2243 to remove each member individually.
Shaves 8-bits off of each IR op.
No need to cart around this data when it is constant for each operation.
Especially since most optimization passes don't need the data anyway.
Needed to add a new `GetRAArgs` to get the number of SSA arguments that
get RA versus `GetArgs` which returns all SSA arguments the IR operation
owns. This is what was causing #2243 to fail CI since it needs to know
the difference in some places.
string::find() returns npos (-1) if the given character was not found, so
it can't be used to construct a string like this. Luckily, the use of
std::span allows this code to be written such that it's both correct and
simpler than before.
Somewhere during the refactoring/review process, failed to strip the
shebang prefix off of the arguments.
Causing shebang files to always fail as if the file never existed.
Fixes steam execution.
In large blocks we can be generating a ton of inline constants. But in
most cases these end up being 0, 1, or (1 << N).
Add these to a map and reuse if possible. Makes some IR blocks
significantly smaller for later optimization passes.
Fixes#2136
This is a fairly tricky edge case to support with FEX.
If execveat is used with AT_EMPTY_PATH then the application can pass an
FD to execve instead of a filename. This includes FDs that have been
deleted from the disk so the child process can't open it by filename
anymore.
To work around this limitation, we need to pass the FD to the new FEX
process and open it directly, similar to how binfmt_misc works with FDs.
The FD will get passed through environment variables, which the new
process will check for and then remove the variable from the
environment.
Lots of prickly edge cases to support here.
Without binfmt_misc:
- Passes the FD to FEXLoader directly.
- Requires duplicating the FD if it has O_CLOEXEC on the FD.
With binfmt_misc:
- Shebang file, pass directly to FEXLoader, just like without binfmt.
- x86 ELF Files, rely on the kernel's binfmt_misc support here.
- Unsupported ELF files, let kernel handle it through binfmt_misc
Argument handling:
- The application can pass in no arguments.
- Means our application configurations were failing to find a config
- Also various checks in the frontend were failing.
- If opened through an FD, find the symlink for that FD for the
application configuration instead.
Side note:
Fixed a performance issue in execve where when we were checking for file
format support. Either ELF or Shebang files, we were reading the /whole/
file upfront. We only need to read a header worth of ELF files, and only
257 bytes if it is potentially a shebang file. Should dramatically
reduce some application's execve times.
The current separated adr and adrp handlers are difficult to use if you
don't know if the resulting address is going to be within 1MB or 4GB.
Adds a `LongAddressGen` helper that will generate the various pieces of
code that will need to be emitted.
Backward labels:
- Can generate three different code segments depending on distance to
label
- adr if label is within 1MB
- adrp if label is 4K page aligned and within 4GB
- adrp+add if label is within 4GB
Forward labels:
- Can generate three different code segments depending on distance to
label
- nop+adr if label is within 1MB
- nop+adrp if label is 4K page aligned and within 4GB
- adrp+add if label is within 4GB
There is still the limitation that this can't generate addresses to
labels that are >4GB away. Which is fine.
We can do a single LDP upfront when loading from the code cache, which
saves an instruction and one LDP costs the same as a single LDR.
Itty bitty optimization in the hot dispatcher.
Pulled from #2176.
On x86-64 the SA_SIGINFO sa_flag is actually a no-op. It is always used
even if not set.
Ensure that we setup siginfo_t regardless of flag being set.
On 32-bit x86 this still needs to be adhered to.
Little side bits that don't change anything
- EFLAGS is passed in signfo correctly.
- User provided restorer usage locations is documented but not
implemented.
We shouldn't use O_EXCL, since we need to overwrite previous entry PIDs
if they happen to exist. The kernel ensures that PIDs don't overlap, but
in some kernel configurations PIDs are aggressively reused, resulting
in O_EXCL quickly hitting an issue when writing stale files.
Additionally O_DIRECT, this doesn't allow us to write to files, so all
write functions were failing.
Additionally use O_APPEND, we are only ever appending, so let the kernel
know.
Additionally use O_TRUNC, in the case that a stale perf file exists,
this will immediately truncate the file to zero.
These fairly trivially map to AArch64 operations.
CLWB just maps to `dc cvac`
CLFLUSHOPT just maps to `dc civac` without the final `dsb`.
Also captures if something tries using XSAVEOPT without checking.
Optimize `FileLoad` by not using fstream.
For some reason fstream is just really bad.
Switching over to raw pread cuts the amount of time it takes to read
files by a quarter of CPU time.
This is causing some heartburn with the hardlink.
- Removes some termux cmake list hacking.
- Removes the need to do post-install packaging fixups when hardlinks get dropped.
- Removes a custom uninstall target that was necessary before.
Map lookup was quite expensive, switched over to three small vectors
that are constexpr instead.
Some file querying and parsing was fairly slow as well. Optimized to
make that CPU time to go away.
This improves initialization time of CPUIDEmu by 33%
If we have a fixed file to read, we can read it in three syscalls.
fstream is...weirdly slow in the other implementation.
Theoretically in the future this can be improved to a single syscall if
the `readfile` syscall ends up in upstream Linux.
Fixes#2095
std::random_device can fail to find a random source and throw an assert
during initialization.
The constructor allows you to provide a token string to select an
explicit random source, but this is c++ library specific and will still
assert if the source isn't found. Also specific tokens are very much
target and library specific, so it is unsafe to use.
In the libstdc++ case, the default implementation will try to open
`/dev/urandom` which might not be available on all targets.
Instead of relying on this C++ object, use the `getrandom` syscall
directly to generate our RNG used in the ELFCodeLoader.
Resolves any sort of asserting case, blocks on RNG generation, and is
guaranteed to be available since this syscall has been available since a
very old kernel version.
Pulled from #2176.
Ensures that when we are handling signals we are actually restoring a
stack state that is what we expect..
While this could randomly intersect with other stack data, it is highly
unlikely and will still capture incorrect stack frames otherwise.
Keeps it out of release build to ensure we aren't sticking random data
in the stack when it wouldn't have even been checked.
When a process is in the process of forking and getting ready for
execve, it is common practice to do a `close_range` or close loop to
close all file descriptors before the execve.
This is a security/sanitization feature to ensure that FDs aren't leaked
to the child process. While it is more reliable to have these FDs opened
with O_CLOEXEC, people get it wrong all the time so this feature has
been put in place. Both python and glibc wrappers for launching
applications do this.
The problem with this for FEX is that we were using a FILE handle for
emitting JIT symbols to the perf file. When the underlying FD is ripped
out from under the FILE handle, it throws an assert that we can't
recover from.
Switching to a raw FD and checking to ensure the FD is still open on
writes means that we can safely stop JIT symbol logging when a process
is closing FDs under us.
Fixes a crash in Steam early startup where a python script is run for
checking if packages are installed.
Adds three classes:
- Advanced SIMD three same (FP16)
- Advanced SIMD two-register miscellaneous (FP16)
- Advanced SIMD three-register extension
A handful of the three-register extension unit tests are disabled
because the vixl disassembler doesn't support them.
Only six more classes of ASIMD operations remaining once this is merged.
CIVAU does Clean+Invalidate to `Point Of Unification`
CIVAC does Clean+Invalidate to `Point of Coherency`
`Point of Unification` means to L2/L3, so unification of core
visibility.
`Point of Coherency` means SLC/RAM, All cores, DNA engines, etc must be
coherent.
When tinkering I had enabled 16k page support in jemalloc.
This broke pressure-vessel/Proton executing. Back it back down to 4k page size
to fix this.
We'll need to come back to this to see if enabling this can be done
without breaking these projects.
Split off from #2243 to remove each member individually.
IR ops are hardcoded by operation to have a destination or not.
No need to have each operation have a boolean for determining if the
operation has a destination or not.
The number of places things need to know if the operation has
destination or not is better served by using a lookup instead.
All of these operations were only testing positive integers which is why
they didn't show 16-bit failures.
Adds a bunch of negative tests to each ones now that #2314 is merged,
which would have caught them.
Found an application today (hashtree tests) that causes us to spill a
large amount of values on to the stack.
We were encoding larger offsets than what unsigned offset load and store
can handle.
If the offset is too large for the loadstore, use a temporary to put the
offset in to first.
Removes some startup time where we are copying nearly a page worth of
16byte vtable pointers at startup.
Also allows the compiler to choose to inline functions if it wants to.
Every operation that the emitter supports is tested in the unit tests.
For the most part uses vixl's dissassembler and a string comparison to
ensure the emitter is outputing what we expect.
For operations that are PC-relative we instead use a bit-exact test to
ensure it is outputting what we care about. This is because vixl
helpfully outputs the PC that the operation is acting on. Since these
locations aren't static in memory, the PC moves around per test.
Previously this could access more elements than it needs to.
In the case where we pass in a 128-bit vector and perform a 128-bit
operation, only two doubles should be handled, but in this case will
actually try and handle four double elements.
Consider converting doubles within a vector into floats:
e.g. _Vector_FToF(16, 4, Src, 8)
16 is our OpSize
4 is our DestElementSize
Src is our input vector
8 is the SrcElementSize
(OpSize << 1) / Op->SrcElementSize becomes:
(16 << 1) / 8 ->
32 / 8 ->
4
What we actually want it 2 here, we're indexing 128-bit vectors out of
its bounds (not a problem now, since we assume up to 256-bit in the
interpreter).
Similarly with 256-bit vectors
_Vector_FToF(32, 4, Src, 8)
would become:
64 / 8 -> 8
where we actually want 4, since we only have 4 64-bit elements in a
256-bit vector.
This corrects this in the interpreter by just special-casing the 64-bit
calculation.
We generate a /lot/ of block links which causes our shutdown time to
take a while for this map. For short running applications this shutdown
time can take a statistically significant amount of time, and this has
been haunting us for a while.
Using a monotonic buffer resource, we can cut the dellocation time down
to effectively zero on cache clear and shutdown.
For short running applications this basically means we clear their
shutdown time, for long running applications this means a cache clear is
less painful.
I've measured up to 2ms , but more often this takes ~0.5ms before
optimization. Now it is consistently ~0.25ms on a short running
application.
Times will vary /greatly/ depending on how full it has filled this map.
This is useful as a debug option and will be useful to have in upstream
while comparing output between current vixl emitter and the new emitter.
With this in place I can easily do binary comparisons to see where I
have mistakes in the new emitter.
We don't want this enabled in release builds as it is a debug feature.
This has already caught a bunch of mistakes, so make it easier by
upstreaming.
It'll likely be useful in the future as well when we are inspecting code
running in the vixl simulator.
Use one large allocation for all levels of the cache so they are
virtually contiguous.
This allows us to clear the cache entirely by using a single madvise
instead of three. Which ends up being quite a bit nicer.
This was likely an issue with signals racing to the spill handler, which
we have fixed bugs with over the past few months.
This means we don't need to spill all SRA GPR registers anymore, at most
we need to spill three registers that intersect with syscall arguments.
Causes the dispatcher to drop from 4476 bytes down to 3900 for
SVE-256bit supporting targets.
This is done by significantly reducing SVE loadstore ops. Going from 8
instructions per 4 registers, down to 2 instructions.
This is done by switching from 1 register loadstore instructions up to 4
register loadstore instructions. Which should significantly improve
performance on future SVE platforms.
Filling and Spilling to the context is still using the old code path
because SVE doesn't offer non-interleaving loadstores.
Spilling and filling on the stack is fine because we don't need to match
context state.
These should only be accessing a 32-bit or 64-bit portion of memory
depending on single or double precision variants are used. Previously
we'd be doing a full 128-bit load.
We need to swap the second and third results to be consistent with ARM.
Also fixes the mistake where I used vhaddpd instead of vhaddps in the
single-precision 256-bit case.
This will allow us to use this with VEX and non-VEX variants without
needing to insert the upper-lane clearing for 128-bit variants into the
non-VEX path.
That, and this also allows us to not need to add additional template
arguments
32-bit GOT calculation needs to do a call+pop to do get the EIP on
32-bit. LEA doesn't work because it there is no EIP relative ops like on
x86-64.
This causes a terrible block split on every GOT calculation without the
optimization in place.
Now the block can continue through this weird GOT calculation.
This will be worthwhile for our 32-bit thunks where for some reason the
GOT calculation can't be removed. The GOT is calculated even though it
isn't used.
On Fault then ARM will return information about the fault in ESR_EL1 to
the user.
We need to decode what ESR_EL1 means in the context of the fault to get
the flags we care about.
The flags we care about specifically are PF_USER and PF_WRITE.
PF_PROT would have been interesting but I didn't see when this gets
returned to the user. ARM makes the difference if the page is unmapped
or "mapped" with PROT_NONE. x86 doesn't make the distinction here.
This should fix an issue that a user was hitting.
Makes our intent to zero-extend the upper lanes explicit and lets us
remove a special case in the LoadSource implementation.
This also makes things a little nicer since we're not hardcoding 32 byte
stores.
On first FEXInterpreter execution, it is expected that `ConnectToServer`
will fail with `ECONNREFUSED` because FEXServer won't be running.
Skip printing this first messaage to stderr if configured.
If it is some other error message then ensure it is still printed.
`GetEmulatedPath` and `OpenAt` are called a /lot/ in applications.
std::filesystem::path handling here is quite heavy and costly for what
we are trying to achieve.
Remove this usage and instead use lstat, access, and readlink directly
which is a heck of a lot faster.
In particular this helps out pressure-vessel, shaving off launch times
by 1-2 seconds.
Going from ~22 seconds down to ~20 seconds.
`std::filesystem::exists` is particularly gnarly in how it checks to see
if the file exists.
It allocates memory, it creates lists, it splits things, then eventually
checking the status
Remove all this overhead to help out minorly for applications that
execve a lot.
The less number of FPR<->GPR movement instructions the better.
This removes one instance of `ins` and replaces the other with a 64-bit
`dup` instead.
The LoadConstant still turns in to a single `movz` instruction with the
shift.
* BUGFIX:Adding the nullptr check to FinalizeHostTrampolineForGuestFunction. It will lead crash on interpreter code set function nullptr callback. eg. XSetErrorHandler(NULL)
* fix#2189
This causes users pain currently since the JIT isn't doing any caching
and our RA being a hack mess means it is quite slow.
Disable by default to improve JIT time performance.
The Zen+ CI runner doesn't support the UMIP hardware feature, so it
doesn't hit the kernel emulated path.
Instead the instruction returns real data on this hardware. Still in
kernel space, so it is unmapped as expected.
Ran in to this when running Team Sonic Racing.
This game uses Denuvo Anti-Tamper which some versions rely on SGDT.
The Linux kernel catches and emulates this instruction.
It will always return a limit of 0 and a base of `0xFFFFFFFFFFFE0000ULL`
which is guaranteed to be in kernel space.
This gets the game slightly farther but still not running entirely.
This is a bit of a large commit since it is an all or nothing sort of
change.
Instead of doing LoadContext and StoreContext for GPRs and FPRs, any
that are statically allocated (GPR and XMM) should use
{Load,Store}Register directly.
This is now enforced that {Load,Store}Context{,Indexed} will assert if
trying to access these ranges.
This is the first step towards accelerating our JIT compile times in
that it removes the need for the SRA pass to convert all the IR over.
This ensures that from the Dispatcher directly we are handling SRA
accesses as "registers", ensuring that future changes don't need to run
in to this problem.
Even though it wasn't a target of this change, in a simple test
application this removes ~12% of the total JIT compile time.
Two changes here.
- Make the mount path follow server temp folder requirements.
- Will be mounted in `/tmp/` or `$XDG_RUNTIME_DIR/` now
- Switch the FEXServer socket to an "abstract" AF_UNIX socket.
- If the socket is in `/tmp/` then systemd will put the service in a
private `/tmp` folder that only exists for the service.
- If the socket is in `$XDG_RUNTIME_DIR` then pressure-vessel can't
chroot anymore since they make their own runtime directory.
- If it is in `$HOME/.fex-emu/` then it breaks usage where the
filesystem is a mount that doesn't support AF_UNIX like sshfs.
The only reasonable thing to do is to switch over to `abstract` sockets
which will work in all cases.
Tested with pressure-vessel and systemd and now it works in all
situations.
This isn't really a performance issue, more just something that is gross
looking at while looking at unit tests.
Break /usually/ isn't abused heavily by games (Except Denuvo) so not
really a perf concern regardless.
This hasn't ever caused problems and instead just adds a message that
appears in logs for nearly every application invocation.
Remove it because it isn't necessary to track.
Initially I put this in very conservatively to make sure we always clear
out the upper lanes, but since Adv. SIMD operations have zero-extending
behavior when storing results, we can just use a lot of operations as
is, without needing to unnecessarily do the same work twice.
This will be useful for keying specific executables to steamids.
This is sadly required because a bunch of games end up naming themselves
"game.exe" so we can't safely enable thunks for all things shipping a
generic name.
In the case of a platform enabling PrivateTmp then the FEXServer and
FEXInterpreter won't have a tmp folder that shares the socket location.
The runtime directory is a perfect place to share these across
processes.
We use these two IR ops for PSRLQ and PSLLQ respectively, Which is
actually implemented in a quite inefficient way.
Instead switch the ops over to using VExtr which maps directly to one
instruction and can emulate both.
This is the last remaining auxv value that we were missing.
This requires a little bit of setup to match what we are doing in
FEXCore's Dispatcher.
This only tracks how much space is required by the kernel to store its
required state.
When we are installed as a binfmt_misc handler the Linux kernel will
assign us AT_SECURE for setuid binaries.
Ensure we are passing this through for any application that will end up
needing it.
AT_SYSINFO points to the vsyscall location that the kernel provides.
This AUXV value isn't used on x86-64.
If we have VDSO installed then we can use the one provided from there,
otherwise we need to provide a code page.
It seems like some behaviour changed with glibc provided in Ubuntu 22.10
that it now requires AT_SYSINFO.
Fixes wine 7.0 execution inside the Ubuntu 22.10 rootfs.
Segment registers are indexed significantly more than they are changed.
Pay the cost of indexing during the set and store rather than the per
register index.
Should be a fairly significant performance improvement for 32-bit
applications. At least on hardware that doesn't have a data dependent
prefetcher.
Breaks Steam atm and isn't clean.
Fixes a crash that occurs due to `_XInitDisplayLock` due to the display
lock function being initialized to our own handler.
Once XInitThreads is called once then it becomes a no-op.
steamwebhelper was hitting this.
We don't detect the range of the long F80, so we need to set that the
source was in range to fix sin/cos/tan calculations.
If we don't set this flag to zero then glibc will do some additional
operations that causes the value to be incorrect.
Fixes the output of the test application in #2021, probably fixes some
camera orientation problems in games as well.
Chrome seems to like sending us invalid packets of data sometimes. With
an invalid packet type just skip parsing the data entirely.
Fixes an infinite loop in Vampire Survivors.
This was generating GOT prologues even on naked functions which was
breaking VDSO on 32-bit.
Fixes almost every 32-bit application when running with debug options.
Turns out this was calculating OF incorrectly, breaking Denuvo early in
its execution.
Changes the ROL imm OF calculation code as well to be more consistent
and not keep src1 alive longer than it needs to be.
Also adds two new unit tests to ensure this stays correct.
This is a tricky situation that wine-preloader allocates the lower
32MB of stack space through fixed address mmap with MAP_FIXED.
They can't use mmap with an address hint nor MAP_FIXED_NOREPLACE because
it changes behaviour. mmap won't give you the allocation inside the
stack space even if you check `/proc/self/maps` that space isn't yet
allocated. The growable space of the stack blocks those allocations.
So the wine peeps might be SOL if they actually require this allocation
to exist.
To replicate this, allocate the application stack at the same location using an address hint.
This will give us the correct region on a 48-bit VA system, while also
letting it select a different region on a 36-bit VA system.
Allows hardware to pack the move and following destructive operation
together into one constructive operation if possible.
e.g.
movprfx VTMP1.D, VectorLower.D
addp VTMP1.B, Pred, VTMP1.B, VectorUpper.B
is allowed to be merged as if it executed constructively like:
addp VTMP1.B, Pred, VectorLower.B, VectorUpper.B
if the hardware supports it. If it doesn't, then the instructions will
behave like a regular move and destructive addp operation separately.
Previously the config options could only have ever enabled thunks rather than
disable them.
Now sort the code so it can enable thunks, then following configs can
redisable them. Allowing testing with global thunks enabled and
disabling problematic applications.
Also sorts the "ThunkConfigFile" config as lower priority than the
application configs. I wasn't thinking about ordering that hard for
these five configuration paths, but application configs should be higher
priority in this case.
Clang thunks already have these default enabled, but let's also enable
this on the GCC side.
sse2 will enable most things we care about, which matches ASIMD quite
closely.
fpmath=sse removes some x87 usage for 32-bit thunks specifically.
Should effectively be a non-functional-change
Each one of these are sorted through the DefinitionExtracy.py script
running over a temporary header file for each set of includes.
eg:
```bash
$ cat test.h
#include <X11/Xproto.h>
#include <X11/XKBlib.h>
#include <X11/Xlib.h>
#include <X11/Xutil.h>
#include <X11/Xresource.h>
#include <X11/ImUtil.h>
$ ./Scripts/DefinitionExtract.h test.h > out.txt
```
Any custom defined types have been sorted appropriately.
A bunch of missing XKB definitions were missing and added in the
process.
I've had this stashed in my git stash for a while now, I just haven't
cleaned it up.
Fixes a bunch of thunks around X11 applications missing symbols.
X11 has an attribute that was causing function declarations to be
missed.
These definitions exist in XLibint.h
eg:
```cpp
extern void _XEatData(
Display* /* dpy */,
unsigned long /* n */
) _X_COLD;
```
This `_X_COLD` attribute was causing these function definitions to get
missed.
unit tests that show the difference of output between FPREM and FPREM1.
Setup as known failures on everything except for host since we don't
implement fprem correctly.
An incorrect fix to FPREM is as follows:
```diff
--- a/External/FEXCore/Source/Common/SoftFloat.h
+++ b/External/FEXCore/Source/Common/SoftFloat.h
@@ -158,6 +158,10 @@ struct X80SoftFloat {
return Result;
#else
+ BIGFLOAT lhs_ = lhs;
+ BIGFLOAT rhs_ = rhs;
+ BIGFLOAT Result = fmodl(lhs_, rhs_);
+ return Result;
return extF80_rem(lhs, rhs);
#endif
}
```
But we shouldn't implement this fix. We should instead implement a new `extF80_mod`
function that handles the rounding differences between FPREM and FPREM1.
Fixes#2097.
Doesn't attempt to resolve#1538
- Some minor cleanups in the VMARegion struct type.
- Switches over to using the FlexBitSet range scanning, based off this
implementation.
- Move memory region allocation to its own function instead of
constructor
- This will be used by a new constructor later for 48-bit host-side
allocations
- Minor optimization to keep track of Munmap.
- We were burning a bunch of time on backward scanning for free
regions even though we never did a munmap to free anything.
- Now only do backward scanning if a munmap occured.
- Saves a bunch of CPU time
These were currently living in the 64BitAllocator class but can be moved
directly to the FlexBitSet.
Ideally in the future these routines can be optimized so our allocator
is faster but for now these are just moved.
This creates a generic interface that FEXCore can use for timeline
profiling. This allows us to create a generic interface which the
backend details are hidden so we can support multiple timeline profile
APIs.
The only API supported right now is ftrace/gpuvis. Which is extremely
lightweight of an interface with minimal overhead.
We must be careful here since in most cases will will have dozens of
FEX instances running at any given time. So a timeline profiler like
Microprofiler can have major issues since that only ever expects a
single process at a time.
Not enabled by default but just needs the `ENABLE_FEXCORE_PROFILER`
cmake option set to enable.
Allows PMULLOp to function correctly with the amended VPSHUFD entries.
Since this is only used to perform expanded multiplication from 32-bit
entries to 64-bit entries, we can simplify things a little bit.
All we need to do is yank the third 32-bit word down into the second
32-bit word's spot in the vector and let the VUMull/VSMull IR ops handle
it.
Extends VUMull to handle 256-bit vectors.
While we're at it, we can fix a typo in the VPSHUFD called for the
32->64-bit case.
To mirror UMULL, we need to replicate element 0 and 1, not 0 and 2
While we're at it, we can fix this with VSMull as well.
Implements AT_PLATFORM: Ends up being `i686` or `x86_64` depending on
ELF arch
Implements AT_HWCAP and AT_HWCAP2
AT_HWCAP is just CPUID function 01h EDX result
AT_HWCAP2 only has two defined bits in it, which we don't support
either.
Implements AT_RANDOM
Previously we were just sticking hardcoded values in to this.
Now we pass along the host's AT_RANDOM, or we generate our own if that
doesn't exist
Fixes#788
Accidentally didn't implement this for FPREM1 but it /was/ implemented
for FPREM. Fixes an infinite loop in cossin implementations.
Test code from the application returns an incorrect result, but it isn't
due to FPREM1.
```
$ `which wine` ./hello.exe
Sin 1.22460635382238E-16
Cos -1
$ FEXInterpreter `which wine` ./hello.exe
Sin 1.22460635382238E-16
Cos 0.54030230586814
```
Fixes#2021
Due to how we use a modified ABI for these indirect functions, we don't
have a clean way to say that the host_addr lives in a side-argument.
The previous inline asm that moved the value from r11 in to a variable
worked up until you hit functions with 8 or more arguments. At that
point the compiler was generating code before our inline assembly and
using r11 as a temporary, thus destroying our value.
Then a crash would occur and it was very hard to determine why. It would
end up calling some random function (0x1 in this case) from an indirect
call.
This made it /look/ like it was calling an invalid function returned
from the loader but in reality it was a corrupt register loading bad
data.
To work around this case, we can use an inline asm register variable and
a volatile asm block that "sets" the variable. In this case GCC and
Clang both seem to extend the live range of the register from the start
of the function to the use of the variable.
This resolves the issue for now, and I tested quite a large number of
function signatures to see if it would break in the future.
Theoretically our functional testing should catch this, but we don't
currently have something that abuses all the functions like this
currently.
These should be using the real syscalls, not our provided allocators.
While not a problem currently since these redirect to host mmap and
munmap, it will become an issue once we have an allocator that lives
outside of x86-64 space.
The kernel maps the primary ELF with a hint to some place *near* the
middle of the virtual address space. While the interpreter stays at the
top of the address space.
This also adds ASLR to the ELF loading, with a define for debugging and
testing purposes.
Requires both #2013 and #2014 merged first.
Fairly straightforward, just requires enabling lld in this case since
cross-compiling doesn't work well with gnu linker.
Also lld doesn't understand the linker script program header symbolic
names for read/write/execute. So we need to use the raw number there.
Works around an issue where GCC 11 generates broken `init_array` section
and also plt sections that glibc doesn't understand.
The `mov ebp, ecx` was breaking vsyscall and was expected to be used
with the `syscall` instruction rather than `int 0x80`.
Remove that to fix it.
Also remove the pushes and pops around the syscall instruction, these
are unnecessary in an emulated environment, we won't clobber the
registers.
Fixes Steam execution with VDSO.
Specifically fixes /sbin/ldconfig.
Fixes 8df7c2d84f
Fixes Steam launching
I failed to test ELF files that are dynamic with no interpreter here, so
EntryPoint ended up being set to zero, which results in an instant
crash.
Ensure these values are set correctly after the primary ELF and
interpreter are loaded so starting RIP is correct.
Noticed that I forgot to change the Zn register over to VTMP1.
This would have been caught by a vixl internal assert anyway.
Also make the behavior equal with VAddP, where we clear and only copy
over the exact amount of bytes instead of the whole register.
Makes the previously implemented ops a little more self-documenting,
and, if we ever actually need to change this, there's a nice constant
that can be looked up instead of magic 32 values.
This is a duplicate of VDupElement since AArch64 doesn't support an
element extract plus zero of the rest of the register.
Removes and replaces its uses with VDupElement.
This instruction was purely a move that did format reinterpretation.
This was necessary with LLVM when we had implicit IR op sizes.
Now that all vector ops are explicitly sized, this is not only
redundant, but also completely unnecessary sicne we don't support LLVM
anymore.
Remove the op, which technically is a very minor optimization for the
two instructions that still used it.
Since PR #2003, we now set up some predicate registers within
FillStaticRegs. We can now make use of those instead of manually setting
up predicate registers inside the IR opcodes.
Instead of mapping to find a range, unmapping, and then submapping
inside of it.
Map once, then mmap with MAP_FIXED to overwrite the mapping.
This fixes an issue where if you enabled ASAN then it would stick
additional mappings inbetween where we want to map. Thus breaking asan.
Use the fastcall ABI for 32-bit x86 to make our lives easier.
Fastcall ABI puts the first two 32-bit arguments in ECX and EDX
respectively.
Compilers are nice today and allow us to do cross-abi function calls
like this.
_XData32 and _XRead32 don't exist as real functions in 32-bit versions
of these libraries, these end up just being defines that redirect to the
non-suffixed versions of the functions.
Noticed this while tinkering around and is easy enough to solve today.
These IR ops are redundant and mostly unused.
VDupElement does exactly what these operations were already doing and
more closely matches what the hardware wants.
Drops in handling for spilling/filling FPRs using SVE for supporting
AVX.
Also alters the dispatcher and JIT a little to avoid accidentally clobbering
TMP4 (x3 as of this commit)
SVE DUP instruction does sign extension on the incoming immediate, while
ASIMD MOVI does zero extension.
If the immediate doesn't fit then move in to a GPR first and then DUP
from GPR.
The x86 runner had unattended-upgrades accidentally still enabled. It
upgraded a bunch of development packages which broke CI.
Fix the struct verifier so it works with the new packages.
Sadly python3-clang doesn't support all the new CursorKind types so we
need to self-define some of them for now.
Once this tool gets converted over to C++ it will be a non-issue.
This allows to use different toolchain files for each and it reduces
build system repetition in test target setup.
The "tests-32" directories has been integrated into the "tests" one. Tests
that should only run on 32-bit are detected by their filename ending with
".32.cpp" now.
This has a badly coded test that can hang forever. Our timeout kills it
at 5 minutes, which causes it to not even fall down the flake path.
Just disable it outright because of the bad test.
If a test is marked as a flake then it will be tried five times before
giving up.
Works around the problem of needing to babysit CI once a PR is pushed.
As long as we have all the flake tests marked.
Following guidance from cmake's FAQ:
https://gitlab.kitware.com/cmake/community/-/wikis/FAQ#can-i-do-make-uninstall-with-cmake
Due to some of the special handling that we do with installs, we need to
do additional uninstall handling that the install manifest doesn't cover.
Specifically we need to add additional uninstall targets for:
- FEXInterpreter
- binfmt_misc
- guest_thunks (Doing its own uninstall target, so passthrough)
While it isn't generally advised to install and uninstall through source
systems, this is something that users want to do all the time.
This has been asked for a couple of times now.
Fixes#1592
The intent of these annotations was presumably to make it easier to adjust
build settings on a per-test basis, but doing this in the build system is
actually much cleaner.
pidfd_open was added in kernel 5.3 so older kernel devices weren't able
to use `FEXServer -w`. If the syscall doesn't give us an FD to the
process, then use a pipe instead.
Since we are only polling for the FD to hangup this works for us.
Noticed recently that `FEXServer -w` was broken and couldn't understand
why. Turns out that FHU syscall handling was /always/ falling down the
`#else` path in the handlers since cmake `add_definitions` follows
folder scoping rules.
This means it was always returning -1, which was causing FEXServer's
pidfd_open usage to always receive -1, which meant the sendmsg with FD
was always failing, which meant the `FEXServer -w` would forever wait
for a message that was never sent.
Converting the utility over to a target not only fixes definition
scoping problems, but also makes the other paths actually work.
This found some compiling bugs and instead lets us define SYS_pidfd_open
if it doesn't exist. Letting the kernel return the ENOSYS if it doesn't
exist on that platform.
Main thing, fixes FEXServer -w hanging forever.
This is the bare minimum, it only tests glxinfo and vulkaninfo with and
without thunks. Nothing more special than that. Already found the .1 bug
with libvulkan host library loading.
Also in FEXLoader make sure to use `EraseSet` for these runtime options.
Fixes a bug where the config was being set to nothing, breaking the
ThunksDB configuration option.
ASTVisitor is great for iterating over AST nodes by type, but most of our
analysis is based on symbol names. For this task, a lookup in DeclContexts
after parsing is complete is better suited.
Reorder PPA building to be after the github tag. PPA takes a while to
run, so good to get it out of the way up front so it can be handled in
the background while doing the rest of the release.
Also update the link which was renamed.
Termux uses defines for these, so our token pasting fails, but we also
still want to use their define so we can fall down their emulation
library whenever possible.
Prefix an underscore to be able to use both our number definitions and
their defines in the same file.
build folder doesn't exist on a freshly started runner.
Executing the rootfs fetch script doesn't care what the working
directory is. Doesn't need to be in `build/` which doesn't exist on a
fresh runner and will fail chdir.
VDSO is heavily abused by Proton games to the point it is showing up as
CPU time.
Implement a guest-facing only thunk library using the hardcoded VDSO
interface in Thunks.
If available this will always be loaded on application load and set the
auxv value to support it.
This requires a bit of special treatment as our first user of linker
scripts since the format of the ELF must be careful crafted to not break
applications trying to parse it.
This library exposes a handful of symbols:
- clock_gettime
- clock_getres
- gettimeofday
- time
- getcpu
- All previous with `__vdso_` prefix
- LINUX_2.6
All of these symbols get routed directly to the host architecture VDSO
interface if they exist.
AArch64 doesn't have getcpu or time VDSO.
In a microbench, VDSO improved bench times substantially
x86-64 host: 3.612s -> 1.369s - 2.63x speed
AArch64 host: 3.821s -> 2.284s - 1.67x speed
- AArch64 isn't as improved due to missing VDSO symbols
This is also our first /always/ enabled thunk as long as the file exists
x86-64 has five symbols within VDSO that we need to emulate.
Pass this through either glibc or host vdso if the symbol exists.
AArch64 doesn't have the time or getcpu vdso interface, so fall down
glibc instead.
tmpnam is considered insecure since it's vulnerable to TOCTOU issues.
This is not an issue for these tests, but replacing tmpnam is not any
more complicated than silencing the warning.
Kind of sucky that SVE doesn't have a convenient way to manipulate
predicate registers with immediates or anything to make this nicer (that
I know of).
Having to use a temp to clear the upper part of the vector reliably is
bleh.
This avoids the need to provide a fallback definition for platform-specific
macros. The definitions are only added host-side, since only Host.h is
included in any interface files.
Due to how the 64-bit allocator previously worked, it was never subjected to memory
regions larger than 64GB to be tracked. With the change in
PR #1885, this has changed to have regions that will hit sizes larger
than 170TB on some platforms.
Better yet, even with smaller regions it still had a performance issue,
it just wasn't as visible.
First problem: We used MemSet instead of MemClear for the live page
clearing. This caused pages to be claimed as "always in use".
This would cause us to always scan the entire region on allocation, find
that it didn't work and allocate a fresh region on every slab
allocation.
jemalloc saving us here since it allocates slabs from the OS fairly
aggressively.
Second problem: We used MemSet (now changed to MemClear) to "clear" the
state tracking for pages.
This causes ~600MB of memory to be used purely for state tracking.
This was physically backed since we were writing to every bit of
tracking for handling 256TB of VA.
This had a fault dance with the kernel for every new page being hit
here.
Instead of clearing the the bits with a memset, clear it with madvise so
it doesn't consume physical pages at all.
This means we use significantly less physical memory for 32-bit
applications.
With this change, pressure-vessel startup time goes from 24 seconds down
to 17 seconds. 70% of the original startup time.
But really the main savings here comes from the memory reduction that
PR #1885 ballooned, but has been an unseen problem before.
Before that PR we were burning 2MB of physical memory per region for no
reason.
After that PR we were burning up to 600MB of physical memory per region
for no reason. Changing a bit depending on how large the region ended up
being.
This now ends up being 2 pages starting out and grows as more pages are
are used. A significant improvement.
Previously in order to enable thunks, we needed an independent
description file of which thunks to be enabled. This is nice for quickly
testing out new games by setting `FEX_THUNKCONFIG` environment variable.
For users that just want to enable thunks this is an unwieldy
indirection that doesn't make much sense at a glance.
Previously this meant you needed two files as an example:
```
ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat thunks.json
{
"ThunksDB": {
"GL": 1,
"Vulkan": 1
}
}
ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat AppConfig/EnderLiliesSteam-Linux-Shipping.json
{
"Config": {
"ThunkConfig":"~\/.fex-emu\/thunks.json"
}
}
```
Instead of this unwieldy redirection just support `ThunksDB` json
directly in the AppConfig.
```
ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat AppConfig/EnderLiliesSteam-Linux-Shipping.json
{
"Config": {
<...>
},
"ThunksDB": {
"GL": 1,
"Vulkan": 1
}
}
```
As can be seen this makes this significantly easier for new users
getting in to thunks. Depending on which path to enable thunks the user
is more comfortable with, they can still enable them using the
`ThunkConfig` option or embedding directly in the application
configuration.
Additionally this removes the older non-ThunksDB path to loading thunks.
All users of it have moved on to using ThunksDB.
We have separate configurations for the Application path versus the
application name we are using as a configuration choice.
Example 1: FEXBash "wine Crysis64.exe"
Previous APP_FILENAME will contain `/usr/bin/wine`, which is still used
elsewhere.
This new APP_CONFIG_NAME will contain `Crysis64.exe`
Example 2: FEXBash glxgears
Previous APP_FILENAME will contain `/usr/bin/glxgears`
APP_CONFIG_NAME will contain `glxgears`
We didn't have this exposed any other way before.
Found an issue with wine + DXVK + thunks where these were passing in
more than 7 arguments and crashing.
Create some assembly to support any size of variadic stack packing.
Only implemented for AArch64 for now.
While the default value of `git config core.abbrev` is `7` (used to
truncate the commit hash in several git commands) and most people don't
configure anything else, I happen to prefer when commits hashes stay
valid for a while, so I changed that value to `20`. Because of this,
FEX fails to build in External/FEXCore/Source/Interface/Core/CPUID.cpp:961
Let's be explicit in the git command about what minimum length of commit
hash we expect.
For these unit tests we no longer need to put them in the disabled tests
file. Instead it will be skipped if the host doesn't support the feature
required.
This is necessary for the fexserver to function correctly when chrooting
in to our rootfs and doing things.
Requires independent rootfs script modifications which will come with
the next rootfs update.
Problem comes down to a chroot supporting multiple users, where our
typical use case is only one user. Bind the server file to a single
server for the entire chroot session regardless of users, solving this
problem inside the chroot.
Fixes apt-get inside of chroot, which runs as user _apt.
Arm64 doesn't have the classic getdents syscall, only getdents64.
Old glibc versions (like 2.17) don't support getdents64
This fixes 64-bit ls with a centos 7 rootfs. Likely also fixes some other very
old applications.
Packing differences between 32-bit and 64-bit getdents means we need to
template this between the two types, otherwise it's quite similar.
No known applications rely on the 32-bit getdents but worked with test
applications.
Some of these functions are missing that games rely on.
Only glGetVkProcAddrNV will cause us pain, so avoid it for now.
The rest are simple and at least glXWaitX fixes some games.
Fixes#1893Fixes#1893
Multiple games will query symbols that are leaked but we don't support.
It is safe to return nullptr in these cases.
Print a warning message still just in-case someone fails hard at GL.
This changes how host trampolines for guest functions are created. Instead
of doing this purely on the guest-side, it's either the host-side that
creates them in a single step *or* a cooperative two-step initialization
process must be used. In the latter, trampolines are allocated and partially
initialized on the guest and must be finalized on the host before use.
Depending on the operation we will do a vector insert or removal while
iterating over the vector.
Fixes a use after free that asan found when insert caused the vector to
resize.
This will help compiling on older distros which are shipping older
Vulkan-Headers.
We need to catch newer Vulkan features earlier than what distros ship
since it is highly common that users will update their drivers through
means that make their drivers be newer.
For example to build on Ubuntu 20.04 we will support symbols much newer
than what that verison of the distro supports.
We could wrap all uses of newer features behind `#ifdef` checks, or
include the newest version of the loader that FEX itself supports.
The submodule is much cleaner and resolves the issue of drivers
supporting newer vulkan versions than libvulkan-dev.
Again super common with Nvidia blob users, kisak-ppa users, or people
updating mesa directly to be on the bleeding edge.
Due to glibc issues around static applications doing dlopen this is a
fundamentally broken option and no longer supported by FEX.
Remove the option entirely as to not be confusing.
We kept this around initially for chroot support, but with our RootFS
mounting AArch64 folders inside the chroot this isn't necessary anymore.
Steam's gameoverlayrenderer.so relies on libX11 symbols to be available
without actually loading that library directly. This works on an unthunked
system since libGL.so depends on libGLX.so, which in turn pulls in libX11.so
at load-time. Adding a fake libX11 dependency to the libGL-guest thunks
reproduces this behavior.
Currently FEXBash only outputs `FEXBash>` which gives weird docker
vibes. This can confuse users since they no longer see what folder they
are in.
Change this so it still has the user and path exposed like a typical PS1
eg: `FEXBash-ryanh@ryanh-TR2:/mnt/Work/Work/work/FEXNew/Build>`
If the user doesn't have any of the tools necessary for handling FEX's
images then the tool would spuriously fail with `Couldn't parse rootfs definition URL.`
With zero indication as to why we removed images from the parsed json.
If the user has at least one of these tools installed then they won't
get this error message.
This function must be able to handle both guest heap pointers *and* host heap
pointers, so it only forwards to the native host library for the latter.
This is because Xlibint users allocate memory using internal macros aliasing
to libc's malloc but then they free using the function XFree. For libX11,
this is not a problem since the allocation happens in a thunked API function
(and hence on the host heap), but if a function from an unthunked library
accesses Xlibint, it will allocate on the guest heap.
One notable example where this was encountered is XF86VidModeGetAllModeLines.
Necessary for tests that depend on the state of the running context.
Since we support an SSE mode and an AVX mode, the FPR store truncate
test will fail on hosts that don't support AVX as the register offsets
are going to be different between the two. So we can conditionally
enable support for these tests.
In the event that the host doesn't support the requirements for running
AVX-enabled applications (SVE2 with at least 256-bit wide vectors) but
still wanted to run regular SSE-enabled applications, they would be
taking a performance hit due to a load/store pessimization (necessary in
order for 256-bit loads/stores to work)
However, we can add an alternate view into the xmm data that would allow
those hosts to use the previous optimization, while still supporting
AVX-capable hosts.
In most cases we aren't using JIT symbols but still creating the perf
map file.
Early check if we should generate the file or not, this way we stop
polluting the /tmp folder.
It's common for Linux applications that use close_range to pass in ~0 as
the last FD. This was causing FEX to spin from [2, ~0U] in this loop.
This would take /forever/ to run.
Change over to an ordered map and use the map's range searching and
ranged erase to more quickly remove these elements.
Fixes a hang that occurs with first time Steam setup in
steam-linux-runtime heavy application `steam-runtime-identify-library-abi`.
Removes the "Load Default Options" menu option. This option was
confusing for new users and isn't necessary anymore.
Fixes the "Load Default" option so it actually populates the full
configuration layer in the face of partial configuration.
This is a /very/ common use case for new users that ran through
FEXRootFSFetcher, where the only configuration set is the RootFS.
The configuration would be visually confusing since the visual
representation for missing options wouldn't reflect their default
configuration state. "TSO Enabled" is an example where it would appear
disabled in the GUI, but it is default enabled.
Also fixes the issue that the default configuration window would just be
a 320x240 floating window in the center of the screen. This is due to
the window being a floating sub window in the dockspace by default, and
not docked.
Instead just remove the dockspace, it isn't serving us any purpose.
This means the child configuration window now maximizes to the window
size which is the desired behaviour from default.
Additionally only save the config file once. While the msg dialog is
open (2 seconds while it is open, or escape to make it go away
immediately) the program won't save the file again. This fixes an issue
that if you used the shortcut key to save the file, it would save the
file at the refresh rate of your screen. Which is 144hz on my setup, so
it spams my filesystem quite heavily.
Main options here are `-y` and `-a`.
Passing both will allow the user to download the exact match image, use
it as-is, and set the config to use the rootfs by default.
Additional options `--distro-name` and `--distro-option` allows one to
select a particular distro version.
eg: `FEXRootFSFetcher -y -a --distro-name=ubuntu --distro-version=22.04`
* Will download the first Ubuntu 22.04 compressed in the json list
** Priority of erofs or squashfs depends on order in json and what the
host supports
While we were getting the application name for the application layer, we
were failing to store the filename for telemetry.
Save the filename we get for application layers and store it for the
telemetry file.
Otherwise these were just alway ending up as wine or wine-preloader.
glibc lazily initializes the SETXID signal handler until first thread
creation.
Once we create our first pthread, steal it back from GLIBC after the
fact.
Fixes SOMA again.
We were registering syscall handlers to the temporary working vectors for
both x86 and x86_64 regardless of which bitness we launched with.
We then only installed the syscall handlers depending on bitness.
This was burning a decent amount of time and some memory on
initialization. Instead just don't register syscall handlers on the
other bitness.
Also stop wasting memory, clear the syscall registration vector after
we've registered everything.
Saves about 28KB of memory per process.
Even though this member isn't used on ARM, the member is still there.
Fixes crashing in optimized builds and also makes unity's garbage
collector significantly more stable
x86 has six instructions that will fault on us that we mostly handled. A
few of these weren't being handled correctly.
One problem is that RIP needs to synchronize differently depending on
which fault instruction it is. Some instructions fault at the
instruction RIP, some at the instruction afterwards.
Additionally some of the metadata generated around the signal delegation
wasn't correct.
With behaviour of all of these instructions changed, it will now be
easier to switch over to non-faulting guest synchronous signals off of
these instructions. This doesn't go far enough to change that behaviour
yet.
I noticed this when looking at Elden Ring's weird faulting behaviour
with ud2 and `int 0x2d`. This made me investigate since I had a
suspicion that we weren't handling both cases correctly.
With this change, Elden Ring is now stabilized and works under FEX.
This is a very basic test to check for overflow exception.
`into` instruction only exists on 32-bit, so it needs to not be built as
a 64-bit executable.
This comes in the form of a new define and a new function.
Assuming assert doesn't quite cover 100% of our asserting logging use
cases so we need to break this out.
If the compiler can't see through side effects at compile time for the
predicate then it'll throw a warning.
This gives us a nice behaviour. In the case that asserts are enabled, we
fall down the assert checking path. So if an assert fails to predicate
we will do an assert log like normal.
The change comes when we build release without asserts, we use the
`__builtin_assume` definition to allow the compiler to optimize around
assumptions. Since these aren't runtime asserts in release build, this
gives us some small optimizations in various locations.
Clang will very specifically do additional optimizations when you
provide it `__builtin_assume` directives that can really be worth it.
Allows for a smaller struct size.
This movement of the struct members also requires us to modify the
DeadContextStorePass to take the new locations into account.
On the plus side, we get to remove all of the padding bits in the
CPUState struct, so we can remove the handling for them.
Now that we have the AVX config option in place, we can use it to
determine how to set up the stack for storing and loading all of the
necessary state we need.
If we have SVE2 support and a vector length of at least 256, then we can
adequately handle AVX.
However we also add in the ability for application profiles to disable
AVX if necessary for any reason.
Not currently used, but will be in subsequent changes.
On Linux the bottom 12 bytes of the legacy FXSAVE context are used to
encode information about any extension blocks that may follow it.
We can use the same approach to encode our high lanes of the AVX
YMM registers into memory and vice-versa.
These require the previous commit's RIP block synchronization.
This is due to the fact that these rely on exceptions to occur on
try-catch blocks.
With us ensuring RIP synchronization on block entry, this solves the
problem of these crashing before doing anything.
Wine 6.x didn't require this, but 7.x does.
Fixes Proton Experimental hangs when it is installing the VC runtimes.
If an application is doing long jumps on exceptions then we need to
ensure that RIP is synchronized at least to block entry for some amount
of safety.
Due to our block linking which doesn't ensure that RIP is synchronized
on block entry, our exception handling wouldn't see a RIP change, thus
not going down the path that we jump to Dispatcher loop top on RIP
change.
Burn a couple of instructions on block entry to ensure that RIP
synchronized but only on config.
When some SRA code was being shuffled around, this config variable was
never being set. This caused the guest signal handlers to never expect
SRA to be enabled.
Moves the Dispatcher config object to the actual dispatcher rather than
having it in the arch specific dispatcher class. Then have the signal
delegator check the config directly instead of having this secondary
config value.
Fixes vc_redist_x64 and vc_redist_x86. Probably also fixes a bunch of
other random Proton/Wine things that rely on signal long jump.
The recursive algorithm used here previously led to deeply nested function
calls, which eventually exhausted the available stack space. The simple
non-recursive algorithm used now avoids this problem at the expense of
small overhead.
Dota Underlords hit this when querying Vulkan two different symbols
that resolve to the same function. Ignoring the error in that specific
case is safe, since the linked guest functions have the same implementation.
Since we have switched over to thunking the vulkan loader, behaviour has
changed here and we need to thunk this library.
While a bit unsafe to thunk arbitrary libraries, we know this one is
safe to thunk.
Fixes thunking Vulkan on steam games which have been broken since
switching over to vulkan loader thunking.
In the future this may become unnecessary but it is required for now.
Currently thunk prefixes are mutally exclusive. No thunks database path
can currently have more than one prefix. If this is necessary then we
can add it in the future.
Most minor of optimizations here, we only scan the string once to find a
prefix, instead of searching for it on all four prefix replacements.
std::string_view was sticking around for longer than libraries being
loaded.
This was causing a crash.
Change this to a std::string directly until we support cleanly removing
this data on library unload.
Fixes a bug in `ExecAndWaitForResponse` where results > 1024 bytes would
overwrite data.
Switches from a custom format txt file to a json file.
JSON file now has a "Type" field to specify squashfs versus erofs.
JSON is now versioned so we don't need to move the file around, just
append to a new versioned segment.
Only shows erofs files if you have the bleeding edge `erofsfuse`
application.
This application was available starting with erofs-utils v1.5 which was
released on 2022-06-13, so it isn't available pretty much everywhere.
Check for qemu and box binfmt_misc file conflicts before the
`binfmt_misc` install command.
This ensures if you're building from source that you won't inadvertently
install conflicting binfmt_misc files, breaking program execution.
Instead of relying on runner features, classify based on CPU features.
This fixes an annoying issue where if running unit tests locally without
it set then you get an unexpected failure.
Fixes#1807
This was checking for the wrong debug structure and it wasn't actually
unlinking it correctly from the linked list.
Since it was talking directly to a_0 instead of the current modifying
struct.
Instead of having a stubbed debug report struct, we can just remove the
structure from the linked list. Since we are already abusing a const
cast there anyway.
Fixes Vulkan thunks on Snapdragon.
When wine-preloader is executed it doesn't do an execve to passed in
wine program. It will instead map the executable directly in to memory
and start executing it.
This way we end up with a program executing like `wine-preloader
<absolute wine path> Game.exe`
This now handles the wine-preloader case so we can get the correct
application profile here.
pressure-vessel overrides our rootfs when it does a pivot_root.
Since we are still communicating to the FEXServer we were pulling the
configured rootfs.
Instead check if we are in pressure vessel and avoid doing that.
This fixes FEX running under pressure-vessel.
(There may be some implications to this down the road with code caching
but let's worry about that later)
It can be useful to know in tooling when the current active FEXServer
has exited.
Two things can happen when this command is run.
No FEXServer is active, returns immediately.
A FEXServer is active, we query for a pidfd from the active server, then
we wait until it exits.
Both instances of this is valid to use.
Interface definitions must enable this functionality by enclosing functions
that may be called through function pointers in a namespace annotated with
`fexgen::indirect_guest_calls`. The guest thunk must further link any host
function pointers to a guest-side instance of CallHostThunkFromRuntimePointer.
We weren't adjusting the guest stack size when using clone3.
If the clone comes from CLONE3 then we need to offset the RSP by the
provided stack size.
This also translates to fork/vfork through clone, so make sure to adjust
stack in that case as well.
Fixes Ender Lilies crashing with Ubuntu 22.04 rootfs
This is a relatively invasive change since multiple things needed to
happen at once.
* Socket based logging is removed
* Logging has been replaced to only support stdout, stderr, and server
* Server is now default and replaces what FEXLogServer did
* Server logging now uses a pipe instead of a socket
* Can be faster than stderr and stdout since the application doesn't
need to wait on terminal output
* FEXMountDaemon has been removed
* Functionality has been merged in to FEXServer
* FEXServer is always executed on FEX initialization time
* Similar in behaviour to Wine's wineserver
* Can explicitly start this before using FEX for logging purposes
* Stays around until all instances of FEX exit
* Will stick around for a short amount of time in case of spurious
execution
* FEXServer will soon be extended to do more than logging and squashfs
mounting
* FEX rootfs scripts will need to be updated to support this path
* Just means rbinding the /tmp folder and forcing a FEXServer instance
to be alive
* Pressure-vessel works fine in this case since FEXServer will already
be running
* It already rbinds the host /tmp folder which is why this works
DRM ioctls are taking advantage of the fact that any ioctl structure
that is passed in to the kernel smaller than the expected value will
zero out the remaining members of the structure.
Some more ioctls coming down the pipe will also abuse this, so we might
as well as get started with the v3d ioctl that requires it.
Due to how this works, the kernel knows how big an ioctl structure
should be and if the userspace passes in a smaller struct, it will
memset the remaining size to zero.
std::filesystem::canonical is very heavyweight and walks the full path
to ensure that each folder in the path is not a symlink.
eg:
```
readlink("/proc", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/proc/self", "880556", 1023) = 6
readlink("/proc/880556", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/proc/880556/fd", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/proc/880556/fd/5", "/home/ryanh/.fex-emu/RootFS/Ubuntu_22_04/usr/lib/x86_64-linux-gnu/ld-linux-x86-6"..., 1023) = 86
readlink("/home", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh/.fex-emu", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh/.fex-emu/RootFS", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh/.fex-emu/RootFS/Ubuntu_22_04", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh/.fex-emu/RootFS/Ubuntu_22_04/usr", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh/.fex-emu/RootFS/Ubuntu_22_04/usr/lib", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh/.fex-emu/RootFS/Ubuntu_22_04/usr/lib/x86_64-linux-gnu", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/ryanh/.fex-emu/RootFS/Ubuntu_22_04/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2", 0x7ffd5646e210, 1023) = -1 EINVAL (Invalid argument)
```
This is what was occuring for every single mmap that occurs. /really/
adding to the time for the syscall to take.
This also happens on a couple of other syscalls which are using this new
path now.
The primary reason why this works is that we know that every entry in
`/proc/self/fd/` is a symlink. So instead of asking for canonical, we
can just read the symlink and this will redirect us to the canonical
path.
So this previous example goes from 14 syscalls down to 1.
eg:
```
readlinkat(AT_FDCWD, "/proc/self/fd/5", "/home/ryanh/.fex-emu/RootFS/Ubuntu_22_04/usr/lib/x86_64-linux-gnu/ld-linux-x86-6"..., 4096) = 86
```
While this is only a minor improvement in the "typical" operating environment,
this significantly improves performance of FEX under proot or if the
rootfs lives on a network share.
Wine will set the application name later in the boot process but we
can't defer application loading that late.
Once an application is loaded with wine or wine64, then check the next
argument for the application name instead.
This will allow us to have wine application application profiles.
eg: FEXInterpreter `which wine` $HOME/.wine/drive_c/GOG\ Games/Oblivion/Oblivion.exe
This will give us the application name of `Oblivion.exe`
Same with: FEXInterpreter `which wine` C:\\GOG\ Games\\Oblivion\\Oblivion.exe
Gets rid of some magic numbers and reduces the number of things that
need to manually change (e.g. when supporting AVX and needing to
increase the xmm size).
Instead of duplicating prefixes in the ThunksDB json file even more,
just do a prefix replacement when the thunks database is being parsed.
Changes the JSON overlay arrays over to @PREFIX@ and reduce the
duplication.
Adds support for the pressure-vessel prefix `/usr/lib/pressure-vessel/overrides/lib`
This gets thunks ready for running under pressure-vessel, once vulkan
thunking switches from device libraries to the vulkan loader it should
just work.
32-bit allocations were scanning for available pages by shifting by the
length of allocation. This is incorrect since large allocations would
then only scan through the region in very large chunks. Especially if
MAP_32BIT was present. Instead scan by page size to ensure better fit.
This fixes X-Plane 11.
Also ensure we don't try to munmap a range unless the result is
MAP_FAILED, noticed we were trying to munmap ~0ULL
The final tail of the file reading was incorrect, so our hashing was
"correct" but it was using stale data from the previous block size read.
Noticed this while wiring up the CI rootfs fetching since the hashing is
a lot simpler there.
Now instead of reading a tail, just attempt to read the full block size
and use the resulting data size instead. Confirmed it matches expected
results now.
In the process we are going to need to update hyperlinks and hashes
anyway, change the hash to XXH3 so it is faster to run.
open is rarely used compared to openat and openat2, so this has been
just missed but nothing really got upset about it.
Fixes Proton Experimental while running under pressure-vessel.
This was originally set to zero out of concern for any application that
is doing anti-emulation, anti-cheat, anti-VM checks.
This concern is likely unwarranted, and if any application/game starts
hitting this as a problem then we can throw an application profile at it
instead.
This makes pressure-vessel emulation checking more optimal by it not
having to do a uname dance.
The latest ARMv8.0 toolchain is implementing fetch_add with a bic rather
than an and. Not sure why they started doing this but support the
remaining logical operations in our unaligned atomics handler.
Fixes the Interpreter ARMv8.0 atomic ops.
Fixes#1742
Steam's webhelper has started enabling its sandbox which completely
breaks under FEX since we don't support seccomp.
Curiously the Chromium code is actually supposed to support a fallback
namespace only mode, which is used in glibc 2.34 environments.
The startup script for this will try to use this namespace only mode,
but Chromium developers never tested this in an environment that doesn't
support seccomp.
Due to an early check in their sandbox code, it checks for bpf support
before checking for which sandbox mode it is entering. This returns
early with a false statement which brings the entire browser instance
down with an assert.
Inject the --no-sandbox argument so we get around this and the sandbox
is disabled.
The pause instruction is architecturally defined to be the `REP NOP`
instruction.
This allows people to use this instruction as a backwards compatible
pause without checking for CPUID support. In fact there is no way to
check if the hardware implements this as a `REP NOP` or a `PAUSE`.
If you have new enough hardware then this just ends up being a PAUSE.
Pass this PAUSE over to our host to help out applications that are
writing spin loops with a PAUSE in it, which we were deleting
previously.
This is an IR op that produces nor consumes any SSA values, but has side
effects.
Turns in to the pause instruction on x86 and yield instruction on
AArch64.
Previously if the instruction was encoded to use rsp, rbp, rsi, or rdi
then due to how these were encoded in modrm this would hit the frontend
path for writing to the high 8 bits of a 16bit register.
This is because it's instruction specific if an 8-bit modrm instruction
chooses to use the high 8-bit region or the upper 4 registers.
See the ModRM.reg section of `ModRM.reg and .r/m Field Encodings`
specifically to see what each encoding stands for. Has four different
meanings per encoding depending on instruction.
This instruction doesn't actually write to registers at 8-bit size, it
extracts an element at 8-bit size and then zero extends it to the full
GPR.
When storing to memory it always stores to memory at the size of the
element extracted.
I grepped around the instruction tables to see if there were any other
instances of this mistake. This was the only one.
Fixes#1472 and also gets Psychonauts 2 running.
Fixes a regression in fa87c73b9ee60a334eace2cdc3097725cbaf5b88; curl complains "curl: option -: is unknown" when trying to fetch a RootFS without this.
std::erase_if shouldn't allow you to modify the object passed in to the
predicate.
libstdc++ hasn't always enforced this but now it does with libstdc++12
clone2 stack pointer passed in points to the highest address for the
stack.
clone3 switches this around and gives us a base pointer and a size.
glibc started using clone3 for its thread cloning which finally caught
this bug. Necessary to run any application under the Ubuntu 22.04 rootfs
since that uses a new enough glibc to encounter this.
This does the setup for handling the named region object loading and
closing using the async interface.
This exercises the async interface while the async thread itself only
does the minimum no-op steps required to fake loading and saving.
The no-op interface is hooked up to the point of exercising it in the
most minimal of sense.
If the configuration is set to enable read-only or read/write object
code then it will spin up the async worker thread as well, but it
doesn't do anything yet.
x86 has eight instructions that are non-temporal.
Only one of which is a load-NT.
Adds a memory access type classification to our LoadSource/StoreResult
helpers.
This lets us explicitly choose Default, TSO, NonTSO, and Stream.
Stream currently just behaves like NonTSO so at some point in the future
we can add non-temporal loadstores to the IR.
Main thing is to move these NT accesses to non-TSO.
This class is similar to std::shared_mutex except it is safe for the
same thread to increment or decrement the ref counter multiple times.
This can be passed to regular std locks.
This will be required with the code object cache service soon.
Documentation claims that these insert the lower 32-bits leaving the
upper bits unaffected.
Hardware testing proves that the upper 32-bits of the base registers are
zero'd.
Additional documentation also concurs that this is the case.
This allows us to have RCPC loadstore operations with a 9-bit signed
offset.
This gives us a small range of [-256,256) of immediate encoding range on
our TSO loadstore operations.
Updates the inline constant pass in ConstProp to support this range on
TSO IR ops if the host supports RCPC2.
Apple M1 supports this extension, didn't test with Cortex-X2/A710.
The JIT currently doesn't use this. This is just the handling code
itself.
One line disabled handling Guest RIP move relocations until the JIT
object cache is enabled.
The JIT currently doesn't use this. This is just the handling code
itself.
One line disabled handling Guest RIP move relocations until the JIT
Object cache is enabled.
It wasn't using acquire semantics, only release semantics.
This was causing the swap to load data from a stale cacheline, causing
the futex system in glibc to break.
This break only occured if you tried going down the RCPC codepath
because of edge case memory ordering problems.
This then enables the RCPC code path now since it works.
In a large number of cases we are moving pointers within a 4GB region
and some marginal pointers that are within 1MB.
This is only used in the case that MOVZ can't be used.
NOP padding still occurs after these instructions to ensure that if they
are being used with relocations it will still get padded to a full 4
instruction length.
Not all hardware fuses these and LLVM claims that Cortex beyond A72 even
doesn't, but it'll still be faster.
Creates a pool allocator for OpcodeDispatcher and IRCompaction that
shares memory allocations between threads in a pool and supports
reclaiming stale allocations from participating threads.
A thread will use a heuristic to keep its claimed memory allocation
around if it is allocating a lot of code. If it slows down then it will
start putting the memory allocation back in to the thread pool.
Additionally if the allocation has been "disowned" and gone to sleep
while still retaining the allocation, then another thread can inspect
these stale allocations and reclaim it from the idling thread. Saving
further memory.
This needs some more work and cleanup but this is an interesting concept
that saves a decent amount of memory even in a basic test.
Causes teeworlds' title screen to go from 754MB to 599MB in my simple
test. 79.4% the memory usage is a good start.
This requires us to pass in --no-rc to bash since otherwise PS1 gets
overwritten by shell variables and nothing happens.
Which this is fine for the common use case of just wanting to run a
basic bash script under emulation.
Since we are masking signals before compiling code, we no longer will
receive a signal in the middle of compiling code.
This makes the compile service never be invoked so we can just remove
it.
We still have some locations in the syscall handling that isn't signal
safe, but compileservice wouldn't have fixed those anyway.
For 128-bit divides, we can very quickly check at runtime if we can
avoid the long divide and just do a 64-bit divide.
For unsigned just check if the top bits are all zero.
For signed just check if the top bits match bit 63 of the lower bits.
Additionally, keep the long divide handlers inside of the dispatcher.
This keeps the majority of the code bloat out of the code block itself,
significantly reducing block size for something doing these divides.
Also a fairly large icache improvement from this.
Hard performance number improvements here are hard to get since it
heavily depends on the application, also only occurs on x86-64.
Seems to have helped FTL and Dead Cells performance quite a bit though.
Exposes the host architecture through this CPUID function. Only exposes
the architectures we support. Not burning 16-bits on using ELF machine
definitions here.
Uses 4 bits still for future expansion.
If the host doesn't support the instructions required for implementing
an instruction then don't even add them to the opcodedispatcher.
This means that we will never try emitting instructions that the host
doesn't support (For these instructions anyway) and successfully passes
the guest SIGILL for these particular instructions.
Fixes#1631
Was incorrectly setting the FCW to 037h when it was supposed to be
037Fh.
Fixes a bug in a visual novel where its CPUID state wouldn't initialize
if this was set incorrectly.
For the JIT cores we don't need to keep IR around, it's only necessary
for the Interpreter. So once the AOT IR service is done dealing with the
IR, check to see if we can delete it.
This causes teeworld's title screen memory usage to go from 730MB to
566MB. 77.5% the memory usage there.
This is effectively an infinite memory leak if the codespace wasn't ever
overwritten or invalidated. So larger memory usage programs would end up
having a larger impact.
I misread the implementation details of this instruction when
implementing.
The pseudocode says `ST(0) = ST(0) ∗ 2^rndint(ST(1))` so I understood
the instruction to use the current rounding mode of the host to extract
the integer portion of `ST(1)`.
The actual implementation is in the details of the statement `the
integer portion of the floating- point value in ST(1).`
This behaves like round towards zero/truncate, additional hardware
testing and documentation reading confirms this.
Fixes#1584
memory-map is not something we want to use. Adds a comment about it and
disables it.
Also changes core events to wait for an event from GDBStub for waking up
which fixes a hang.
We need to fetch the full list of map files from the memory map and hand
it over to gdb.
It will then fetch all the libraries from the remote host and give us
backtraces
These are static per process and only need to be initialized once.
We are going to support multiple signal handlers from the backend after
this, so can only install once.
I accidentally committed a 32-bit range that was significantly smaller
than what it should be.
While the minimal range worked for simple cases, it didn't work for
anything complex.
Give it the full range it needs.
Fixes#1600
In the case of an AArch64 builder is using 16kb or 64kb pages like is
common on servers then it would fail to compile, even if the resulting
application would only ever run on 4k page hosts.
Resolve this by removing the build check and hardcoding 4kb pages for
each of our uses. We still require 4kb pages to run, so this mostly just
removes the weirdness where it is 16kb builder + 4k runner. Would have
broken some of our assumptions when running.
FEX hardly works on Termux as-is, but we should make sure to document
how to update the packages otherwise we will quickly become outdated on
their package management.
The documentation is incorrect about this instruction. It claims that
you use 66h prefix to choose between operating at 8B or 16B.
This is incorrect, real hardware only responds to REX.W for choosing the
operating size. These other prefixes are ignored but is still accepted as
an instruction decoding.
This matches the AArch64 implementation fairly well.
Bundles RDRAND and RDSEED together for simplification, both instructions
are a single flag on AArch64.
This greatly simplifies the IR format by using string parsing for
gathering the information.
Tons of redundant information removed.
Significantly more difficult to mess up adding a new IR op.
Significantly improves the generator functions in IREmitter
This region is reserved for hypervisor uses. Let's follow other examples
and return a hypervisor vendor id signature as another way for software
to find if it is running under FEX-Emu.
According to PEP 386: https://www.python.org/dev/peps/pep-0386/
distutils is deprecated and will be removed in an upcoming python
version.
Switch over to pkg_resources for version parsing and comparison
The Android filesystem they are on just doesn't support them
Instead of hardlinking FEXLoader to FEXInterpter, just build the
executable twice and eat the filesystem cost.
In some cases we can generate more optimal code if we have more
information about a syscall which number gets const-propagated.
In particular optimizing through syscalls, not synchronizing state, and
never returning.
- Noreturn is used by a syscall that never returns, like exit.
This means that it never needs to try and synchronize state coming back
- Not synchronizing state and optimizing through syscalls
Useful for syscalls that don't read the state past arguments and only
returns a value.
This picks up all the instruction implementations, including 3DNow!
Extended and the Geode specific instructions that were added.
Most of these match preexisting SSE instructions except that they
operate at 64-bit and in the MMX registers.
We already decoded the modrm `rm` bits, check while decoding modrm to
ensure we don't try decoding it again.
Was causing double decoding of SIB and displacement bytes, breaking
things
In the face of an application using MAP_FIXED_NOREPLACE AND the host
linux kernel doesn't understand this flag. Then we were falling down the
hint allocation path which would allocate a pointer in 64-bit space,
returning this pointer to a 32-bit userspace and breaking things.
Now when the hint fails with this flag, we know that it intersecting a
range and can early exit.
Allows some something like the following:
"Env": {
"FEX_MAXINST": "500"
}
Not that I would recommend overriding MAXINST in the asm tests, as
command line overrides that
This allows x87 fallbacks to be loaded from the upcoming code cache
without relocations.
Only 40 pointers necessary to store and means x87 code won't hit
relocations heavily.
Probably improves performance slightly on the x86 host side but should
be neglible.
Needs #1574 and #1575 merged first.
Just like the previous AArch64 JIT.
These pointers are process or thread specific depending on the pointer
and should be loaded from the State object.
Performance here might slightly increase.
This is required for code cache on x86
Needs #1574 merged first.
These pointers are process or thread specific depending on which pointer
it is.
All of these pointers end up getting used inside of the JIT blocks
themselves and with code caching would result in a ton of relocations
occuring inside the code.
The pointers used within the dispatcher don't currently matter since I'm
not expecting to cache the dispatcher itself. It's only a page per
thread after all. This does move us significantly closer towards using a
single dispatcher for all threads though.
The performance impact of this change is unlikely to be felt at all,
some locations have less code generation which could improve perf
slightly. Some locations move from a 1-3 cycle constant calculation to a
4 cycle load, hard to be felt since it gets hidden by other
instructions.
x86-64 JIT will be added soon after this
This is pulled from the code cache PR. Will be necessary for supporting
relocations.
Not currently being used but will be once we have code caching in place.
This is just a memory leak waiting to happen.
Only the primary thread in an application really should have this set
since the kernel cleans it up.
We only ever allocate the primary thread of the guest application then
every host thread's stack on top of that. It's up to the guest when it
is cloning to set up new stack pointers, we don't manage that.
We are already allocating the first thread's size at the soft stack
limit with RLIMIT_STACK anyway.
Fixes#1556
Fixes executing musl applications with the dynamic linker.
It was using the main executable's p_offset instead of the
interpreter's.
Wasn't a problem with glibc since it uses a different symbol to find the
base (Don't ask me why it does this).
musl dynamic linker on the other hand just uses AT_BASE directly and
since it was calculated incorrectly it was crashing.
Testing application was `ls` which had a p_offset of 0x40, so it would
try and read some values from AT_BASE, starting at an offset below where
it was mapped.
The run_thunkgen* helpers now parse generated source code and return its AST
representation, so HasASTMatching helper calls don't each need to redundantly
compile it themselves. This also ensures the generator output actually compiles
in tests where we didn't explicitly check that before.
This also allows printing the full AST of the generator output on test
failures. This must be enabled manually by changing a variable in the ostream
output operator for SourceWithAST.
Previously, defining zero-initialized variables right before LOAD_LIB
could cause the compiler to put thunk definitions into bss, hence triggering
errors during assembly ("attempt to store non-zero value in section `.bss'").
Host thunk libraries are always built as part of the main project now.
Guest thunk libraries are still cross-compiled in a CMake ExternalProject,
but *additionally* there are CMake targets in the main project to make
sure IDE engines can properly handle guest source files.
Fixes an issue where the FEXRootFSFetcher wouldn't get a any user input
and just fail out.
Save it to the tmp folder and execute from there instead.
Fixes#1557
FEX always builds with PIE and we don't hit this issue anymore anyway.
If some application wants to inject a page in to the lower 32-bits then
we have no reason to complain about it anymore. Just let it go and
hopefully they know what they are doing.
Fixes#1559
Just noticed this while casually reading the x86 architecture manuals.
The move segment registers instructions ignore the REX.R prefix on the
segment register.
Previously this was expected to create an invalid register selection.
A little bit silly but sure, support it.
Some build environments use a define to change stat64 and statfs64 to be
the same definition as stat and statfs.
Check if the define exists and if it does then remove the 64bit
constructors.
The host provided siginfo_t definition can vary depending on the build
environment.
What doesn't change however is how the data is laid out.
It's always 128bytes, The first three 32-bit words are always known.
The 64-bit host side always has an additional 32-bit pad member.
Then the remaining bytes is the sifields.
The instruction decode tables for crc32 introduced some dumb.
`F2h` and `F2h && 66h` prefixes both work for crc32.
This is a failure on Intel's part for sticking crc32 in to the vector
table.
MOVBE without any prefixes also does the same garbage where prefix `66h`
acts as an operand prefix size ONLY.
This table is particularly terrible. CRC32 is the first instruction in
this table that needs either prefix `72h` OR `66h && F2h`
For 8bit CRC32, this ignores the 66h operand size override prefix.
- But our table decoding didn't handle this
For 16bit/32bit/64bit CRC32 this behaviour changes depending on 66h
prefix AND REX.W
- 66h prefix is ignored when REX.W is set, always 64bit but it falls
down the other table path
This is an absolutely weird edge case that nobody should hit, but here
we are.
This was mainly an optimization around memory usage. ALU ops tend to
bloat the IR quite heavily, but I also noticed a 2-4% uplift in
performance of some applications. So a nice side effect.
Should let us more aggressively target reducing our IR intrusive
allocator size since this is quite reduced.
In a pedantic heavy ALU op code block this reduces the number of IR ops
from 14,756 IR ops to 2,016 prior to optimization.
After optimization both had reduced down to 50 IR ops, proving the
output IR was the same.
Dynamically linking xxhash is causing problems with pressure-vessel.
With this in place we only have the typical C++ dependencies
```
$ ldd ./Bin/FEXLoader
linux-vdso.so.1 (0x00007fff44d9d000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4c4d884000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4c4d7a0000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4c4d786000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4c4d55e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4c4e0fa000)
```
By default we won't build with the interpeter to reduce user confusion.
The interpreter isn't really useful to end users so remove it.
Completely removes it from building except for the fallback operations.
This also removes the selection from FEXConfig to remove selection
confusion there.
File Stats:
FEXLoader Size with Interpreter: 3422768 bytes
FEXLoader Size without Interpreter: 3301944 bytes
Size difference: 96.4699915%
Bytes removed: 120824 bytes
4k pages removed: 29.498046875 -> 30 rounded up
VM Stats (Reported from bloaty):
Memory Size with Interpreter: 6.50Mi
Memory Size without Interpreter: 6.38Mi
Size difference: 98.1538462%
If an application is forking heavily with threaded file accesses
happening then the mutex can end up in an unknown state.
On fork make sure to lock the mutex then immediately unlock after fork
occurs.
This final step resolves hanging that pressure-vessel hits on startup.
Since it is doing a ton of file opening and forking during
initialization.
Instead of just a basic mutex, also mask the signals.
This fixes the problem where we can end up receiving a signal in the
middle of memory allocation. Thus leaving the locked mutex in a broken
state.
This more closely matches the Linux kernel behaviour.
Since if you're in the middle of a memory allocating syscall, you won't
get signaled.
Currently force disabled until the rest of SSE 4.2 is enabled
This is to remind us in the future that SSE4.2 can only be enabled in
CPUID with CRC32 instruction support.
This isn't correct and breaks games.
This makes the FREM and REM1 implementation the same.
While not 100% correct, it is still better than before.
New issues will be created to handle the differences in the future.
Fixes#1374.
Also fixes most of the HL2 issues, just not the seam issue.
Some of these behaviours have changed now, particularly around signal
handling.
Some things still fail now of course. But most everything is now
documented as to why it is failing or disabled.
Fixes#955
On Set, we have four options that need to be converted.
On Get, we have two options that need to be converted.
This fixes a crash that Tomb Raider 2013 was having on launch.
Makes curl do its continue feature to give the users the best chance of
downloading a rootfs. We don't need to restart the full file transfer on
failure. Helps people with slower connections.
On failure to download, asks the user if they want to retry the download
rather than just exiting with a weird error about hash failure.
Once the image is downloaded, now changes options depending on if
squashfuse or unsquashfs works.
Prevents the user from selecting a bad option and getting unexpected
behaviour. Ideally we would do a squashfs mount test as well for
platforms that don't have working FUSE, like termux. This is harder to
get right and its for an unsupported platform, so I'm not going to
invest more time with it.
Fixes#1525Fixes#1526Fixes#1527
Location to check if curl, squashfuse, and unsquashfs are working.
unsquashfs is a bit more complex where it needs to parse the help output
to see if zstd is supported
In the case of launching without stdout/stderr then redirection could
have these constants be a redirected FD that sits in the same fd number.
Use -2 to indicate no redirection.
Use -1 to indicate closing traditional stderr/stdout
The rest will indicate if stdout and stderr should be replaced as
normal.
Making sure not to close the incoming fds if they matched the
stdout/stderr FD numbers.
This cropped up mostly with multiblock and `jmp <signed displacement>`
This also happened with non multiblock `jcc <signed displacement>`
Due to how IR relocations occur, this needs to happen fairly late but
isn't a big deal.
Fixes#1517
Brings along a bunch of enhancements and ensures we always build against
the latest version.
Also fixes up a few issues that arose due to changes in fmt
Needs to be a string_view for the MIDR for the StrConv helper to work in
this instance.
There is no null terminator character when reading from the file is why.
AssertHandler by default synchronizes but MsgHandler with Assert level
should also synchronize.
Fixes an issue where LogMan::Msg::AFmt wasn't syncing so the
FEXLogServer would never see the messages.
cmake will bake in the environment variable in to the build scripts.
Instead have the guest_test_runner fetch it at runtime.
This means if you forget to set ROOTFS prior to running cmake, you can
now set it afterwards and rerun with just ctest instead of a cmake
dance.
Fixes#315
In the case that multiple messages appearing in a single packet then we
were repeating the first message and dropping any subsequent messages.
Fixes#1496
Before doing anything that requires curl, actually check if it is
installed.
Then instruct the user to install curl before using.
Doesn't try installing curl itself since we don't have a clean way to
execute sudo from potentially GUI.
Fixes#1498
If the user passes in an absolute path then check to see if it exists in
the rootfs before executing.
Useful for launching applications directly out of the rootfs with
FEXInterpreter.
In the case that the absolute path doesn't exist in the rootfs then
fallback to the host system as usual
Some syscalls support `AT_SYMLINK_NOFOLLOW` In these instances we need
to follow the symlink on a couple of syscalls.
Fixes executing wine using the basic wine path
eg:
FEXBash "wine dxcapsviewer.exe"
I forgot about this option working for tuning arch on AArch64. This will
be used in PPA releases in the future. Will leave the previous option
since it can be used in testing.
This is definitely a bit divisive but overall this is a win.
This is a user pattern that is emerging in a bunch of projects.
Allow an officially sourced script that lets you pipe a script directly
in to python/bash and setup the environment entirely.
This only supports Ubuntu {20.04, 21.04, 21.10, 22.04} which matches
exactly what we expose in the PPA.
Once this is in the repo, and our PPA is updated to the latest release
tag you can run this script like:
`curl --silent <Direct Github raw link> | python3`
Once the PPA is updated, the README and Wiki will be updated with Quick
Start guides to use this path.
The script steps
1) Checks if ARMv8, or fail
2) Checks if supported Ubuntu, or fail
3) Checks if PPA installed
3a) Install PPA if not, or fail
4) Check if packages are installed
4a) Install non-installed packages, or fail
5) Check if RootFS is configured/exists
5a) Run through FEXRootFSFetcher to get/setup rootfs, or fail
6) Attempt to run emulated uname -a through FEX, or fail
7) Provide some examples for how to use FEX
8) Exit with success!
Run() is expected to be used after a pause at this point.
Make RunUntilExit() notify using the StartRunning variable entirely.
This is because if we used Run() then the initial thread hasn't yet set
up its core or signal handlers. So it'll ignore SIG63. Then its reason
for pause is still set to Resume.
Once an application tries sending SIG63, we try restoring the context
from a paused state that does exist and crash
Due to the way this syscall handles siginfo_t, we only need to shift
data through the padding.
This is due to the syscall not allowing you to send siginfo_t with
si_code being positive. Which is what the kernel uses to interpret the
data if it is reinterpreting it.
This message can actually cause a perfectly running application to crash
due to signaling at the wrong time.
Instead constexpr disable it so if you want to see it. You can still
enable it
This is intended to be used by a package maintainer to override the
version when the git repo or git executable isn't available.
Not expected to be used by normal users. Instead by our automated ppa
tooling.
When our FEX version string has less than 16 characters this would
result in size_t underflow, resulting in a crash.
This only ever occurs on a release tag so it went uncaught at first.
The PPA builder managed to uncover this problem since it only deals with
release tags.
By default we tune to the native CPU, using some heuristics to determine
the true native CPU since Apple doesn't expose an MIDR.
Adds an option that allows the user to pass in the CPU to tune for.
This will be useful for debugging and also for package building.
I thought this was incorrect but since it only supports IPC_64 it is
actually correct.
If a 32-bit application wants to use the old encoding then it needs to
use the ipc syscall instead.
Fixes#1253
This instruction is only available on 32-bit x86. On overflow flag set,
this instruction will raise a SIGSEGV with overflow exception set in
siginfo_t's si_code.
This implements the everything required to raise the signal and modify
our signal results that we are giving to the guest.
This is effectively the first step in generating signal frames with
custom data in it but only enough for INTO today.
In the case of having a stale lock file on the system, usually due to an
unclean shutdown, have FEX retry if the socket for that lock file is
also not available.
This resolves the exact case of:
```
FEXBash xeyes &
kill -9 `pidof FEXMountDaemon`
sudo shutdown -r now
FEXBash xeyes & <--- This command would now fail
```
The first command would do:
- Check lock file and socket
- Spin up FEXMountDaemon
- Create Lock file and Socket file
- Start executing XEyes in FEXInterpreter
The second command would do:
- Kills the FEXMountDaemon process
- This leaves squashfuse running
- The lock file and socket file are now unmanaged and just files on the filesystem
The third command would do:
- unmounts the squashfuse mount
- Leaving a dangling folder as a mount point
The fourth command would do:
- Check for lock file and socket
- Attempt to use lock file, socket, and mount that is dangling
- Fail with error
Now instead fourth command will do:
- Check for lock file and socket
- Check if filesystem is valid
- Knows that lock file exists but socket isn't active at all
- Deletes lock file
- Spin up FEXMountDaemon
- Create Lock File and Socket File
- Start executing XEyes in FEXInterpreter
The main one that we can't implement is readdir. Falls in to the same
problem space as getdents.
With this, we have the full entry tables filled out for both 64-bit and
32-bit. With some holes in the implementation, we have almost all
coverage now.
In the case of nothing being set then with the default being zero now we
will not have fixed it up to calculate the number of threads based on
host core count.
Adds a header only include utility folder that can be included from
everywhere.
Contains syscall helpers for older glibc and defines for older Linux
uapi headers missing some defines.
This tool supports both a zenity and tty interface.
TTY will be presented if available while Zenity will be used otherwise.
This tool pulls a rootfs list from https://rootfs.fex-emu.org/
It then allows you to select a rootfs from the list, download it, place
it in to the correct working folder, extract it if desired, and set it
as the current default RootFS.
This requires curl and potentially zenity installed to use.
Maybe also unsquashfs if the user chooses to extract the image.
This is an all in one tool to quickly get a new user up and running.
Additionally if you pass in a file path in to the tool, it will generate
an xxhash of the file and exit out. This is the hash used to ensure the
files are valid.
Fixes thread, memory-map, and OS data packet types.
These were attempting to substr when the encode function already handles
that.
Was making it so gdb was only ever receiving the first 1000 bytes of the
data and then decoding incorrectly.
On x86 this is always supported.
On ARM this is only supported if FPCR writes actually enable the things.
Also detects the AFP feature for flushing input denormals to zero.
These are all part of the x86 MXCSR.
No Cortex supports FPCR exceptions, while Apple M1 CPUs support
Exceptions but not the true "AFP" extension
Apple instead supports some additional flags in their
`SYS_APL_AFPCR_EL0` register for enabling this.
If the CPU is unknown inside of the ARM CPU detection then the
MIDROption selected could have fallen down a path where it is set to
nullptr.
Resolve this crash by doing a nullptr check.
We were dereferencing the shmun ptr instead of using it directly.
Resulting in an almost immediate crash
Additionally IPC_SET doesn't write back to the shmid_ds provided.
Additional still SHM_STATE/SHM_STAT_ANY/IPC_STAT wasn't writing its
result back to the guest. Resulting in invalid stat information for the
guest.
Additionally SHM_INFO doesn't follow IPC64 behaviour since there is no
shm_info 64-bit type for a 32-bit OS.
The kernel just truncates the results in this case. Could be an
oversight on the kernel dev's side?
Fixes#1252
For the x86-64 JIT this is implemented with pulling rdtscp's result for
this value.
For Interpreter and AArch64 JIT this is implemented with the getcpu
syscall.
Theoretically AArch64 could implement this with MPIDR_EL1 but because
SoC vendors hecked this up, we can't. Thanks.
Kernel just returns zero + reserved bits if you try reading it.
Syscalls on x86-64 have a gap in the range of [335, 424) where these are
defined as returning ENOSYS.
This was a decision that the Linux developers decided on so x86-64 can
realign its syscall numbers to a common infrastructure.
Turns out that Proton Experimental was using a 32-bit only syscall to
check if the feature existed and was listening for ENOSYS.
They did this unconditionally on both x86 and x86-64 and it hits this
gap section.
This was introduced in Proton Experimental with commit
563fb0fbe2a49734aede87bedccb20daefc553b0
Message: "ntdll: Use clock_gettime64 if supported."
This is technically valid since it falls within this gap section but
leaves a bad taste.
We can safely ignore this section so set it up with the
UnimplementedSyscallSafe handler and also assign it to Syscall MAX so it
can go down our fast handler.
Not that this is likely to be a performance critical path.
Fixes Proton Experimental crashing when run under FEX.
Some of these were in the Emitter class and some were in the
HostFeatures.
Merge these together since in the future I'm going to be using all of
this data as a key for our AOT code cache.
Still only setting the new state if RIP is affected for now.
Noticed a bug where we weren't setting our frame RIP to the new RIP on
32-bit.
Decided to walk through more of the state setting while fixing that.
This gets #1214 further but then it eventually crashes with a read to
0x11.
On FEXMountDaemon startup, fetch the FD limit and current number of open
files. This allows us to track the number of open FDs we have.
Once we get close the the safe threshold we then try increasing the soft
limit towards the hard limit. If we bump up to the max (Say someone
setting the hard limit to 256) then print some fairly big warning
messages.
With the previously fixed FD leak, we could very quickly hit the pipe
limit because Steam is constantly opening new processes all the time.
With the leak fixed, running Steam and idling, FEXMountDaemon only has
around 16 pipes being watched.
This is another edge case where if the process was sent a SIGTERM then
it can get in to a weird state where pipe tracking is broken.
We already check for zero pipes being tracked at the end of this loop,
just check for the shutdown variable to be set instead
Even though we were removing the pipe from the epoll interest list,
we were failing to close the pipe fd. This was leaking the FD and
running in to the Linux process FD limitation over time.
This fixes a class of issues where if you were running FEXMountDaemon
that hit the max, then there were FEX processes not being watched. Which
means that FEXMountDaemon could exit while FEX was still trying to run.
Usually a non-issue since squashfuse would keep running in the
background since open files in the mount were still in use.
Could hit fun race conditions because of it though.
On an overburdened system then FEXMountDaemon may notrespond within the two
seconds of FEX waiting for an ack.
In this case then we hit a bad edge case where we overwrite the lock
file with a new FEXMountDaemon instance then spin up a new
FEXMountDaemon.
This new FEXMountDaemon will recognize that one is still running and
early exit, but now that we've overwritten the lock file with a new
location, all future FEX instances won't be able to find the squashfs
rootfs.
Instead, ignore that the ack wasn't received and attempt running
instead.
FEXMountDaemon should still pick up the pipe message and monitor FEX
regardless.
This fixes the `/usr not found` messages and crashes resulting from it.
Each section map invocation was clearing the sections pointer, meaning
we only ever had one section in the vector.
We want to keep all of the sections around from the ELFCodeLoader.
Fixes a bug where we were effectively never catching any code segments
from the executable passed in.
This allows the compiler back these tables into the executable, which
reduces the amount of work the function has to do at runtime.
Reduces the runtime of this function by 30% relative to the previous commit.
The tables have recently been changed to be zeroed out as a whole on startup.
Speeds up InstallDebugInfo by about two orders of magnitude and reduces Debug
executable size by 1.8%.
Notably, xcb_take_socket uses the callback_guest annotation, since the
given callback is never called on the host (instead it's manually forwarded
back to a helper guest thread for calling).
Function pointer arguments given to the guest thunk library aren't
callable on the host, so they need special handling on a case-by-case
basis. Similar problems arise when the guest-side tries to consume a
pointer returned from a native host library. To automate some common
scenarios, this change adds two new annotations:
"callback_guest" indicates the callback parameter is never called on the
host and hence can be marshalled like any other argument. Accidental host
calls to the function pointer are prevented by wrapping it in an opaque
type alias.
"returns_guest_pointer" indicates that the host function returns a pointer
usable in the guest context. This applies e.g. to functions that derive
the returned pointer from input guest pointer arguments.
Some applications set callbacks that never get called in practice (such as
error handlers). It's sensible to just not implement these instead of
cluttering the code with effectively unused callback wrappers.
Splits out the few required dependencies to a FEXCore_Base static
library.
FEXCore then links to this directly.
Then make it so the FEX Common code links to FEXCore_Base so the
jemalloc dependency doesn't get pulled in.
Had some idle time so I implemented this logic.
We do some tricky logic to have a big.little configuration even with
unknown CPU core types. Promoting or demoting a single MIDR depending on
if we have a mixed configuration or not.
In a non-hybrid design we only claim product names inside the CPUID
product string.
This will appear if you `/proc/cpuinfo` or read the CPUID registers
directly
eg on Snapdragon 888:
processor : 0
model name : FEX-2112-1-g13b14b85 Cortex-A55
processor : 1
model name : FEX-2112-1-g13b14b85 Cortex-A55
processor : 2
model name : FEX-2112-1-g13b14b85 Cortex-A55
processor : 3
model name : FEX-2112-1-g13b14b85 Cortex-A55
processor : 4
model name : FEX-2112-1-g13b14b85 Cortex-A78
processor : 5
model name : FEX-2112-1-g13b14b85 Cortex-A78
processor : 6
model name : FEX-2112-1-g13b14b85 Cortex-A78
processor : 7
model name : FEX-2112-1-g13b14b85 Cortex-X1
eg on Macbook Pro VM which can't see the CPU type:
processor : 0
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
processor : 1
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
processor : 2
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
processor : 3
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
processor : 4
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
processor : 5
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
processor : 6
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
processor : 7
model name : FEX-2112-1-g13b14b85 Unknown ARM CPU
vixl has an assert check to ensure the register sizes are the same for
ubfm.
32-bit Inline syscalls hit this for both arguments and return value.
VCastFromGPR would have hit this but there aren't any x86 instructions
that move 8-bit and 16-bit values in to a vector register.
Several API functions act as state querying functions. These can take
some parameters by const to communicate that we don't intend to modify
the respective passed in instance.
.at(0) can blow up in the case we receive a message from user code with msg_iovlen=0
.data() will simply return a pointer to the empty buffer and we can
proceed with forwarding the call to the syscall.
As a continuation to #1404 these unit tests were needing to be added.
This tests both select and conditional branch merging for integer ops
and float ops.
This is what discovered the issue fixed in f2de640395
It doesn't make any sense anymore to have specific instruction names
set for UND versus a nullptr string anymore.
Was useful when we could use it to determine the difference between
undefined from the start versus set in the tables but with unknown
decoding. Which is an edge case.
Now instead just zero initialize the data, which means it is an unknown
type and nullptr name. Which works for use.
Improves initialization time of the InitializeInfoTables function from
423 microseconds to 37 microseconds.
Specifically this tries to avoid changing much behaviour and keeping the
code the same. So most of it is a direct transplant without any
modifications. This is step one of the process so I can start logically
separating the code and making sense of it.
This mostly moves the AOT IR handling to its own independent file for
separation. Cleaning up the Core.cpp file quite heavily.
Two minor behaviour changes that got mixed up with this change.
The first one is an ASAN fix.
This is the FEX_PACKED on the RegisterAllocationData class.
I didn't want to change too heavily how this serialization works but I
wanted to resolve the ASAN error. This may change in the coming work.
Problem was the padding betwene the uint32_t and the PhysicalRegister
wasn't initialized but was being read.
Since it is all uint8_t types afterwards there isn't a perf issue here.
Second fix was a crash that occurs if you're attempting to both capture
and load IR on the same run. This is a quirk where we mmap the original
IR file. Then on shutdown the IR file is getting saved.
At which point we open the IR file again, truncate it, and start
serializing all of the IR data.
The truncation makes it so our mmap of the file is no longer resident,
resulting in a crash when reading our IR cache from the mmap region.
Now open a temporary file and rename it after storing.
Resolves the crash but still doesn't really solve the issue of multiple
processes overwriting the same IR files.
First test just ensures that an aligned cacheline clear will clear out
the data.
Second test ensures an unaligned address will still only clear the
cacheline the address lies in without touching the other surrounding
cachelines.
Disabled on the host runner since the CI host runner doesn't support the
instruction
This instruction zeroes a cacheline in memory that is weakly ordered and
non-temporal.
It uses the RAX register for where in memory to clear and aligns the
address on cacheline regardless of actual alignment.
This zeroes out an emulated 64byte cacheline. Writing zeros to memory.
This very specifically is only 64bytes to match x86 behaviour.
Also specifically non-temporal and weakly ordered. Which matches x86
CLZero behaviour.
try_emplace() will return the existing entry in the map if it
exists already, so we don't need to perform a find() and then
try_emplace(). We can just use try_emplace() by itself.
Given the calls are all internal, we can store the function pointers
directly and call them with the this pointer instead of indirecting
through std::bind and std::function.
Without this, the internal BucketList type will always be allocated with
T as a uint32_t, due to the default type for T, even if T is specified
differently in other code.
This tool works in both gui-less and gui modes.
FEXLogServer executed alone will open a socket on `localhost:8087` and
pass log output to stderr.
If passed the `-g` argument then it will open the IMGui UI which has
some more options.
It's fairly basic right now but eventually should allow filtering by
PIDs, TIDs, and log level type
Noticed an issue where some applications were thinking that applications
were running on a system with 512 CPU cores.
Specifically freedreno/turnip was trying to create 512 worker threads
which would cause my ARM devices to run out of memory.
Looks like sysconf behaviour has changed around this where it is
actually expecting the uint64_t alignment that the Linux kernel does.
So follow that behaviour as well and resolve the issue.
Adds a new OutputSocket config option that when set will force all logs
to go through a socket.
This allows us to avoid polluting the guest application's output through
a socket instead of a file. Alleviating the issue of a file output
overwriting when multiple applications are ran.
Previously, the host thunk would seemingly succeed to load even when
the corresponding host library could not be loaded, leading to obscure
crashes later throughout execution.
This test doesn't increase coverage significantly, since OP_THUNK is called
with an invalid library name. An ideal test should verify that thunk symbols
are loaded properly, whereas currently it's only ensured the opcode 0xF 0x3F
is recognized by FEX at all. That's better than nothing, but a regression
here would likely show up in other tests anyway.
Instead of early exiting, allow the application to continue running but
throw error messages anyway. Should allow some users to still run FEX
even if an application steals a page in the lower 32-bits
In several parts of the IR we have cases where code accesses header SSA
arguments via:
Op->Header.Args[index]
which can be quite noisy when repeated quite a bit.
This adds member functions to the IR ops that have SSA arguments, that
allow it to simply be:
Op->Args(index)
Namespace-scope variables declared as const or constexpr have internal
linkage by default.
Also makes the output a little more consistent in terms of const/static
ordering.
We can create the table at compile-time, since we already have the
function labels available ahead of time.
This also centralizes the op table in one read-only location
In quite a few places we have a raw primitive to represent an IR node's
ID. This can make reading some bits of the API (or the passes) a little
confusing to take in, since there's no meaningful type name for some
data structure members.
We can provide an alias that communicates this directly to the reader.
This also has the nice benefit of providing a single point of definition
for node IDs which can allow for easier changes in the future (e.g.
making Node IDs strongly-typed etc).
Allows BucketList to work with types that aren't a direct numeric
primitive, so long as the object has equality operators defined and are
default constructible
In the case of storing a thread state without a guest signal then this
would be filled with garbage data.
This would then cause a crash on thread restart.
Only happens using gdbstub
These maps are primarily used for debugging, so we don't need to
construct and allocate the data until GetSyscallName() is called.
Removes a bit of memory usage and a static constructor for release
builds.
Migrates lingering instances of the old logger over to fmt where
applicable. This allows removing some of the old defines and functions.
The only remaining usages of the printf-based variant of the logger is
in Tests/LinuxSyscalls/Syscalls.cpp for the strace handling.
If MAP_32BIT was failing then we weren't checking error result
correctly.
This was causing us to crash.
Now add a helper function for checking syscall results since it is easy
to mess up.
Replaces the few cases where we were manually checking with the helper.
Fixes a Dota: Underlords crash
If set and oldset point to the same location. We need to be careful to
store the old mask before returning it. Otherwise we will will store the
current mask over the new mask and not change anything.
For some reason the full container path isn't actually setup right now.
Until this problem is resolved, keep the FEX rootfs configuration
around.
This lets applications still execute albeit still wrapped with the FEX
rootfs rather than the container's
The CompileService was spinning up with the incoming thread mask and
then setting the mask once running.
Instead set the mask, which the thread inherits, then set it back once
it is created.
Will allow users to choose between a static package or a non-static
package
These packages will conflict with each other, so you can only choose one
or the other.
fex-emu-static: Necessary for Chroots, Can't use thunking.
fex-emu: Necessary for thunking, Can't as easily be used for chroots
If a SIGBUS is received in compile service code then we weren't handling
it correctly. Instead we would fail the JIT space check and hand it off
to the guest.
When compiling code and linking blocks, disable all signals then
reenable once back in the dispatcher.
This adds 15-40 microseconds on to each of these steps and some
additional branching overhead.
This is a problem where our allocator is not reentrant safe and when
receiving a signal we need to jump to a new code region and compile.
Depending on the mutexes that are currently live, we will just stall out
forever.
This is /very/ commonly picked up when attempting to run pressure-vessel
Fixes#1217
Instead of throwing an error and closing down FEX. Instead pass the
SIGILL to the guest application.
On unhandled instruction implementation the instruction, we instead emit
a _Break IR op at that location.
A _Break IR op will ensure the context state is synchronized at the
point of of the fault and has fairly low overhead. We branch to the
dispatcher which does the SRA spilling.
Tested this with an application that attempts an AVX512 instruction,
catches the fault, and continues onward.
With #1383 in place, we also won't pass spurious ERROR_AND_DIE to the guest anymore.
The end of this optimization pass is getting to a bit long.
It might be worth adding a new flag to IR ops soon for ops we can't
optimize context loadstores through.
ERROR_AND_DIE was using __builtin_trap which would send our application
either a SIGILL or SIGTRAP depending on architecture.
This would then be captured by our faulting system and passed over to
the guest application.
If the guest application happened to have a signal handler installed for
these then it would pick up this fault and potentially continue
unsafely.
Now we can remove this usage of __builtin_trap and switch over to our
own handler.
Our frontend will check to see if the fault came from our handler and
uninstall the host signal handlers in this case. Which is what we want
for "ERROR_AND_DIE"
This could also prove useful for guest libraries, however the implicit
dependency of libEGL on libGL (via glXGetProcAddress) cannot be resolved
easily with the current build setup.
By default std::stack uses the heavyweight std::deque container for storage,
but std::vector is perfectly suitable for our purposes and it has better
performance characteristics.
Only the interpreter requires this currently since the stack register
doesn't match.
The JITs correctly set up their stack register and matches what the API
wants.
This is guaranteed to work. signal + rt_sigreturn is a handshake with
the kernel to return the data that the kernel wants.
Only time this doesn't align is when the guest signal handler longjumps
out of the signal handler. Which leaves a dangling entry in the
SignalFrame stack on the interpreter.
Most of these were already marked for passthrough, just needed to be
enabled.
Some syscalls can be passed through but need to be renamed for 32-bit.
This adds another define which does that for us.
These are equivalent to the REX prefix's RXB bits, except that they're
in 1's complement form.
These are trivial to handle and fix usages of the upper range of
registers for ModRM encoded fields.
To ensure that we have coverage for this, I've altered the BEXTR test to
make use of R14 and R15.
We can also make use of data() here to avoid undefined behavior if any
inputs are 0 for whatever reason (unlikely, but still nice to have the
reassurance).
While we're in the area, lets move logging over to fmt to get it out of
the way.
This also gives us an opportunity to merge some CMakeLists things
together to organize it.
In a syscall microbench this improves performance by ~19% on my
Snapdragon 888.
Going from ~9.6 million syscalls per second to ~11.5 million.
Macbook Pro is less effective here due to high syscall overhead due to
VM. Going form 7.2M/s to 7.6M/s, ~6% improvement
We can also inline some 32-bit syscalls but that will need some more
work which isn't done yet. Even though the op in the JIT supports it.
Ran in to this when removing syscall nodes. If an IR op has side-effects
then this generic helper can not remove them.
First time this was encountered and it was confusing
Adds the Arm64 one as well so we can map syscall names directly with
renaming here
Adds some comments about which syscalls have no implementation.
Automatically generated.
These aren't strictly necessary for enum flags and through discussion in
\#1363, would lead to an awkward to use overload.
If these are ever needed, they can be added back at a later date.
Previous we were just using an address hint to emulate MAP_32BIT.
Seemingly this behaviour has changed on AArch64 where it now isn't
guaranteed to scan up from the hint provided if exact allocation fails.
Now we pull in the full 32-bit allocator and add support for MAP_32BIT
in it. This limits the allocations there in to the first 2GB which Linux
expects.
Necessary for Mono's trampolines to work since it requires code to be in
the first 2GB on x86-64.
When the context structure is adjusted in the signal handler then this
updates the state of the CPU on sigreturn.
We check to see if the RIP was adjusted and in this case we will update
the JIT in a potentially unsafe fashion.
This works well enough with signal handlers that effectively just long
jump and not much else.
Fixes wine initial prefix setup where rundl32 expects to do a try-catch
fault capture but infinite loops without this.
With a DYN ELF in most cases the first program header will be at offset
0.
This isn't always true though, in the case of `micro` the first program
header is at a 4MB offset.
This was causing us to miscalculate the size of ELF in memory, which was
causing BRK to intersect with the ELF on allocation.
This would cause ELF loading to fail and then us to close in some cases.
Also makes sure to pass -1 as FD in mmap with anonymous to match.
We were failing to update the emulated signal mask on return from
sigsuspend.
This was breaking Unity titles which was using sigsuspend and then
modifying the signal mask with sigprocaddr
Ubuntu soft links a bunch of binaries in /usr/bin to softlinks that live
in /etc/alternatives/
When hitting any of these alternative softlinks execve would fail if the
host also didn't have the same softlink paths.
Allows us to correctly follow the symlinks on execve as well which fixes
launching wine directly from the wine symlink.
Alternatively you could have launched /usr/bin/wine-stable directly.
Also fixes FEX strace again.
glibc 2.34 changed the select interface to update the timeout on return
to more closely match the kernel interface.
glibc 2.33 always made a copy instead of updating.
Make sure to set the timeout on each iteration of select otherwise we
will end up having a timeout of zero. Thus burning a CPU core.
CPUID lets the application know if it is running on a CPU with hybrid
CPU clusters.
This matches big.little fairly easily. Walk the affinity mask and
check if we are running on a big.little system and report it to the
guest.
For x86-64 host just pass through the flag.
Only a partial fix for #1330, still needs preemption disabled to work.
On x86-64 hosts the Linux kernel resides in the top bit of VA which
isn't mapped in to userspace.
This means that userspace will never receive pointers living with that
top bit set unless you're running a 57bit VA host.
This results in userspace pointers never needing to do the sign
extending pointer canonicalization. But additionally some applications
actually don't understand the pointer canonicalization.
This results in bugs like: https://github.com/golang/go/issues/49405
Now if you're running on a 57bit VA host, this will end up behaving like
FEX but it seems like no one in golang land has really messed with 57bit
VA yet.
In AArch64, when configured with a 48bit VA, the userspace gets the full
48bit VA space and on EL mode switch has the full address range change
to the kernel's 48bit VA.
This means that we will /very/ likely allocate pointers in the high
48bit space since Linux currently allocates top-down.
So behave more like x86-64, hide the top 128TB of memory space from the
guest before boot.
Testing: Took the M1Max 15ms to 21ms allocate the top 128TB.
We've run out of the range of 32 bits already and will need to use
another flag in upcoming changes, so we need to expand our flags to be
64-bit.
While we're at it, we can use a dedicated type alias for the instruction
flags to make the interface changeable from one spot in the future.
We can share this between the interpreter and the JIT. Necessary to
support the TSO-correct interpreter path.
With this change the interpreter is TSO-correct for GPRs. Just not FPRs
yet.
There's zero need for these to have an independent size field and it was
just confusing.
For stores it was always set to zero and for loads it was just
duplicated.
In addition this allows introspection of the store op without casting
the op, which can be useful in edge cases
This is a pretty straightforward operation that can be nicely modeled
by the BIC instruction on ARMv8, which is nice since we can get rid of
the need to manually perform the And and Not operations.
In the case of binfmt_misc being installed, but the user was still using
FEXLoader to pass in arguments then we wouldn't pass the arguments
forward to applications passed through execve.
This resolves an issue where Wine would fail to know where the rootfs
is since Wine launches a bunch of processes.
eg: `FEXLoader -R Ubuntu_21_04 wine winecfg` would fail before
Fixes#1323
If glibc is compiled without static-pie then we can't detect that. We
will just get a compile failure.
Looks like ALARM is compiling glibc without --enable-static-pie for
whatever reason.
Fixes#1326 as much as we can. We need to ask the ALARM maintainers to
change their configuration.
This lets us have JITsymbols grouped by library.
Useful for determining where to thunk.
Sadly perf doesn't have an option to deduplicate regions by name, so
some external tooling is necessary to make it look nice.
Previously the offset mismatch error was expecting two arguments, but
only one was provided.
While we're in the area we can convert the logging type over to the
fmt-capable one which can catch these.
While the previous change eliminated much of the codegen caused by
constructing everything individually on the stack, it didn't eliminate a
memcpy of all the elements onto the stack.
This eliminates the memcpys by allowing the compiler to place all the
data into RO and just reference that data.
These only perform bit arithmetic, so we can allow them to be used in
constexpr contexts.
This allows clang to collapse quite a bit of code for the table
initializing functions. For example, in InitializeVEXTables(),
with these as inline (but not constexpr) functions, clang will
individually put all of the table entries onto the stack.
With these as constexpr functions, clang will be able to deduce that it
can construct the tables at compile time and reduces the amount of
generated code quite a bit.
@@ -18,14 +18,8 @@ This project aims to provide a fast and functional x86-64 emulation library that
* Portable library implementation in order to support easy integration in to applications
### Target Host Architecture
The target host architecture for this library is AArch64. Specifically the ARMv8.1 version or newer.
The CPU IR is designed with AArch64 in mind but there is a desire to run the recompiled code on other architectures as well.
Multiple architecture support is desired for easier bringup and debugging, performance isn't as much of a priority there (ex. x86-64(guest) translated to x86-64(host))
### Not currently goals but will be in the future
* 32bit x86 support
* This will be a desire in the future, but to lower the amount of work required, decided to push this off for now.
* Integration in to WINE
* Later generation of x86-64 instruction sets
* Including AVX, F16C, XOP, FMA, AVX2, etc
The CPU IR is designed with AArch64 in mind but should allow for other architectures as well.
x86-64 host support is available for ease of development, but is not a priority.
### Not desired
* Kernel space emulation
* CPL0-2 emulation
@@ -33,7 +27,3 @@ Multiple architecture support is desired for easier bringup and debugging, perfo
* IRQs
* SVM
* "Cycle Accurate" emulation
### Dependencies
* clang-tidy if you want to ensure the code stays tidy
* cmake
* A C++17 compliant compiler (There are assumptions made about using Clang and LTO)
LogMan::Throw::AFmt(Thread->ThreadManager.GetTID()==FHU::Syscalls::gettid(),"Must be called from owning thread {}, not {}",Thread->ThreadManager.GetTID(),FHU::Syscalls::gettid());
/*FEXCore::ARMEmitter::VReg::v0, FEXCore::ARMEmitter::VReg::v1, FEXCore::ARMEmitter::VReg::v2, FEXCore::ARMEmitter::VReg::v3,*/FEXCore::ARMEmitter::VReg::v4,FEXCore::ARMEmitter::VReg::v5,FEXCore::ARMEmitter::VReg::v6,FEXCore::ARMEmitter::VReg::v7,// FEXCore::ARMEmitter::VReg::v0 ~ FEXCore::ARMEmitter::VReg::v3 are used as temps
LOGMAN_THROW_A(Bytes==0,"Inst at 0x%lx: 0x%04x '%s' Had an instruction of size %d with %d remaining",DecodeInst->PC,DecodeInst->OP,DecodeInst->TableInfo->Name,InstructionSize,Bytes);
LOGMAN_THROW_AA_FMT(Bytes==0,"Inst at 0x{:x}: 0x{:04x} '{}' Had an instruction of size {} with {} remaining",
File diff suppressed because it is too large.
Load diff
Loaded 100 of 1200 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ryan Houdek