Add support for Callee-Saves registers
https://bugs.webkit.org/show_bug.cgi?id=148666
Reviewed by Filip Pizlo.
We save platform callee save registers right below the call frame header,
in the location(s) starting with VirtualRegister 0. This local space is
allocated in the bytecode compiler. This space is the maximum space
needed for the callee registers that the LLInt and baseline JIT use,
rounded up to a stack aligned number of VirtualRegisters.
The LLInt explicitly saves and restores the registers in the macros
preserveCalleeSavesUsedByLLInt and restoreCalleeSavesUsedByLLInt.
The JITs saves and restores callee saves registers by what registers
are included in m_calleeSaveRegisters in the code block.
Added handling of callee save register restoration to exception handling.
The basic flow is when an exception is thrown or one is recognized to
have been generated in C++ code, we save the current state of all
callee save registers to VM::calleeSaveRegistersBuffer. As we unwind
looking for the corresponding catch, we copy the callee saves from call
frames to the same VM::calleeSaveRegistersBuffer. This is done for all
call frames on the stack up to but not including the call frame that has
the corresponding catch block. When we process the catch, we restore
the callee save registers with the contents of VM::calleeSaveRegistersBuffer.
If there isn't a catch, then handleUncaughtException will restore callee
saves before it returns back to the calling C++.
Eliminated callee saves registers as free registers for various thunk
generators as the callee saves may not have been saved by the function
calling the thunk.
Added code to transition callee saves from one VM's format to the another
as part of OSR entry and OSR exit.
Cleaned up the static RegisterSet's including adding one for LLInt and
baseline JIT callee saves and one to be used to allocate local registers
not including the callee saves or other special registers.
Moved ftl/FTLRegisterAtOffset.{cpp,h} to jit/RegisterAtOffset.{cpp,h}.
Factored out the vector of RegisterAtOffsets in ftl/FTLUnwindInfo.{cpp,h}
into a new class in jit/RegisterAtOffsetList.{cpp,h}.
Eliminted UnwindInfo and changed UnwindInfo::parse() into a standalone
function named parseUnwindInfo. That standalone function now returns
the callee saves RegisterAtOffsetList. This is stored in the CodeBlock
and used instead of UnwindInfo.
Turned off register preservation thunks for outgoing calls from FTL
generated code. THey'll be removed in a subsequent patch.
Changed specialized thinks to save and restore the contents of
tagTypeNumberRegister and tagMaskRegister as they can be called by FTL
compiled functions. We materialize those tag registers for the thunk's
use and then restore the prior contents on function exit.
Also removed the arity check fail return thunk since it is now the
caller's responsibility to restore the stack pointer.
Removed saving of callee save registers and materialization of special
tag registers for 64 bit platforms from vmEntryToJavaScript and
vmEntryToNative.
* CMakeLists.txt:
* JavaScriptCore.vcxproj/JavaScriptCore.vcxproj:
* JavaScriptCore.vcxproj/JavaScriptCore.vcxproj.filters:
* JavaScriptCore.xcodeproj/project.pbxproj:
* ftl/FTLJITCode.h:
* ftl/FTLRegisterAtOffset.cpp: Removed.
* ftl/FTLRegisterAtOffset.h: Removed.
* ftl/FTLUnwindInfo.cpp:
(JSC::FTL::parseUnwindInfo):
(JSC::FTL::UnwindInfo::UnwindInfo): Deleted.
(JSC::FTL::UnwindInfo::~UnwindInfo): Deleted.
(JSC::FTL::UnwindInfo::parse): Deleted.
(JSC::FTL::UnwindInfo::dump): Deleted.
(JSC::FTL::UnwindInfo::find): Deleted.
(JSC::FTL::UnwindInfo::indexOf): Deleted.
* ftl/FTLUnwindInfo.h:
(JSC::RegisterAtOffset::dump):
* jit/RegisterAtOffset.cpp: Added.
* jit/RegisterAtOffset.h: Added.
(JSC::RegisterAtOffset::RegisterAtOffset):
(JSC::RegisterAtOffset::operator!):
(JSC::RegisterAtOffset::reg):
(JSC::RegisterAtOffset::offset):
(JSC::RegisterAtOffset::offsetAsIndex):
(JSC::RegisterAtOffset::operator==):
(JSC::RegisterAtOffset::operator<):
(JSC::RegisterAtOffset::getReg):
* jit/RegisterAtOffsetList.cpp: Added.
(JSC::RegisterAtOffsetList::RegisterAtOffsetList):
(JSC::RegisterAtOffsetList::sort):
(JSC::RegisterAtOffsetList::dump):
(JSC::RegisterAtOffsetList::find):
(JSC::RegisterAtOffsetList::indexOf):
* jit/RegisterAtOffsetList.h: Added.
(JSC::RegisterAtOffsetList::clear):
(JSC::RegisterAtOffsetList::size):
(JSC::RegisterAtOffsetList::at):
(JSC::RegisterAtOffsetList::append):
Move and refactored use of FTLRegisterAtOffset to RegisterAtOffset.
Added RegisterAtOffset and RegisterAtOffsetList to build configurations.
Remove FTLRegisterAtOffset files.
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::setUpCallFromFTL):
Turned off FTL register preservation thunks.
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::CodeBlock):
(JSC::CodeBlock::setCalleeSaveRegisters):
(JSC::roundCalleeSaveSpaceAsVirtualRegisters):
(JSC::CodeBlock::llintBaselineCalleeSaveSpaceAsVirtualRegisters):
(JSC::CodeBlock::calleeSaveSpaceAsVirtualRegisters):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::numberOfLLIntBaselineCalleeSaveRegisters):
(JSC::CodeBlock::calleeSaveRegisters):
(JSC::CodeBlock::llintBaselineCalleeSaveSpaceAsVirtualRegisters):
(JSC::CodeBlock::optimizeAfterWarmUp):
(JSC::CodeBlock::numberOfDFGCompiles):
Methods to manage a set of callee save registers. Also to allocate the appropriate
number of VirtualRegisters for callee saves.
* bytecompiler/BytecodeGenerator.cpp:
(JSC::BytecodeGenerator::BytecodeGenerator):
(JSC::BytecodeGenerator::allocateCalleeSaveSpace):
* bytecompiler/BytecodeGenerator.h:
Allocate the appropriate number of VirtualRegisters for callee saves needed by LLInt or baseline JIT.
* dfg/DFGJITCompiler.cpp:
(JSC::DFG::JITCompiler::compileEntry):
(JSC::DFG::JITCompiler::compileSetupRegistersForEntry):
(JSC::DFG::JITCompiler::compileBody):
(JSC::DFG::JITCompiler::compileExceptionHandlers):
(JSC::DFG::JITCompiler::compile):
(JSC::DFG::JITCompiler::compileFunction):
* dfg/DFGJITCompiler.h:
* interpreter/Interpreter.cpp:
(JSC::UnwindFunctor::operator()):
(JSC::UnwindFunctor::copyCalleeSavesToVMCalleeSavesBuffer):
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::usedRegisters):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGStackLayoutPhase.cpp:
(JSC::DFG::StackLayoutPhase::run):
* ftl/FTLCompile.cpp:
(JSC::FTL::fixFunctionBasedOnStackMaps):
(JSC::FTL::compile):
* ftl/FTLLink.cpp:
(JSC::FTL::link):
* ftl/FTLOSRExitCompiler.cpp:
(JSC::FTL::compileStub):
* ftl/FTLThunks.cpp:
(JSC::FTL::osrExitGenerationThunkGenerator):
* jit/ArityCheckFailReturnThunks.cpp: Removed.
* jit/ArityCheckFailReturnThunks.h: Removed.
* jit/JIT.cpp:
(JSC::JIT::emitEnterOptimizationCheck):
(JSC::JIT::privateCompile):
(JSC::JIT::privateCompileExceptionHandlers):
* jit/JITCall32_64.cpp:
(JSC::JIT::emit_op_ret):
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_end):
(JSC::JIT::emit_op_ret):
(JSC::JIT::emit_op_throw):
(JSC::JIT::emit_op_catch):
(JSC::JIT::emit_op_enter):
(JSC::JIT::emitSlow_op_loop_hint):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_end):
(JSC::JIT::emit_op_throw):
(JSC::JIT::emit_op_catch):
* jit/JITOperations.cpp:
* jit/Repatch.cpp:
(JSC::generateByIdStub):
* jit/ThunkGenerators.cpp:
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntSlowPaths.cpp:
(JSC::LLInt::LLINT_SLOW_PATH_DECL):
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
(JSC::throwExceptionFromCallSlowPathGenerator):
(JSC::arityFixupGenerator):
* runtime/CommonSlowPaths.cpp:
(JSC::setupArityCheckData):
* runtime/CommonSlowPaths.h:
(JSC::CommonSlowPaths::arityCheckFor):
Emit code to save and restore callee save registers and materialize tagTypeNumberRegister
and tagMaskRegister.
Handle callee saves when tiering up.
Copy callee saves register contents to VM::calleeSaveRegistersBuffer at beginning of
exception processing.
Process callee save registers in frames when unwinding from an exception.
Restore callee saves register contents from VM::calleeSaveRegistersBuffer on catch.
Use appropriate register set to make sure we don't allocate a callee save register when
compiling a thunk.
Helper to populate tagTypeNumberRegister and tagMaskRegister with the appropriate
constants.
Removed arity fixup return thunks.
* dfg/DFGOSREntry.cpp:
(JSC::DFG::prepareOSREntry):
* dfg/DFGOSRExitCompiler32_64.cpp:
(JSC::DFG::OSRExitCompiler::compileExit):
* dfg/DFGOSRExitCompiler64.cpp:
(JSC::DFG::OSRExitCompiler::compileExit):
* dfg/DFGOSRExitCompilerCommon.cpp:
(JSC::DFG::reifyInlinedCallFrames):
(JSC::DFG::adjustAndJumpToTarget):
Restore callee saves from the DFG and save the appropriate ones for the baseline JIT.
Materialize the tag registers on 64 bit platforms.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitSaveCalleeSavesFor):
(JSC::AssemblyHelpers::emitRestoreCalleeSavesFor):
(JSC::AssemblyHelpers::emitSaveCalleeSaves):
(JSC::AssemblyHelpers::emitRestoreCalleeSaves):
(JSC::AssemblyHelpers::copyCalleeSavesToVMCalleeSavesBuffer):
(JSC::AssemblyHelpers::restoreCalleeSavesFromVMCalleeSavesBuffer):
(JSC::AssemblyHelpers::copyCalleeSavesFromFrameOrRegisterToVMCalleeSavesBuffer):
(JSC::AssemblyHelpers::emitMaterializeTagCheckRegisters):
New helpers to save and restore callee saves as well as materialize the tag registers
contents.
* jit/FPRInfo.h:
* jit/GPRInfo.h:
(JSC::GPRInfo::toRegister):
Updated to include FP callee save registers. Added number of callee saves registers and
cleanup register aliases that collide with callee save registers.
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitGetByValWithCachedId):
(JSC::JIT::emitPutByValWithCachedId):
(JSC::JIT::emit_op_get_by_id):
(JSC::JIT::emit_op_put_by_id):
* jit/JITPropertyAccess32_64.cpp:
(JSC::JIT::emitGetByValWithCachedId):
(JSC::JIT::emitPutByValWithCachedId):
(JSC::JIT::emit_op_get_by_id):
(JSC::JIT::emit_op_put_by_id):
Uses new stubUnavailableRegisters register set to limit what registers are available for
temporaries.
* jit/RegisterSet.cpp:
(JSC::RegisterSet::stubUnavailableRegisters):
(JSC::RegisterSet::calleeSaveRegisters):
(JSC::RegisterSet::llintBaselineCalleeSaveRegisters):
(JSC::RegisterSet::dfgCalleeSaveRegisters):
(JSC::RegisterSet::ftlCalleeSaveRegisters):
* jit/RegisterSet.h:
New register sets with the callee saves used by various tiers as well as one listing registers
not availble to stub code.
* jit/SpecializedThunkJIT.h:
(JSC::SpecializedThunkJIT::SpecializedThunkJIT):
(JSC::SpecializedThunkJIT::loadDoubleArgument):
(JSC::SpecializedThunkJIT::returnJSValue):
(JSC::SpecializedThunkJIT::returnDouble):
(JSC::SpecializedThunkJIT::returnInt32):
(JSC::SpecializedThunkJIT::returnJSCell):
(JSC::SpecializedThunkJIT::callDoubleToDoublePreservingReturn):
(JSC::SpecializedThunkJIT::emitSaveThenMaterializeTagRegisters):
(JSC::SpecializedThunkJIT::emitRestoreSavedTagRegisters):
(JSC::SpecializedThunkJIT::tagReturnAsInt32):
* jit/ThunkGenerators.cpp:
(JSC::nativeForGenerator):
Changed to save and restore existing tag register contents as the may contain other values.
After saving the existing values, we materialize the tag constants.
* jit/TempRegisterSet.h:
(JSC::TempRegisterSet::getFPRByIndex):
(JSC::TempRegisterSet::getFreeFPR):
(JSC::TempRegisterSet::setByIndex):
* offlineasm/arm64.rb:
* offlineasm/registers.rb:
Added methods for floating point registers to support callee save FP registers.
* jit/JITArithmetic32_64.cpp:
(JSC::JIT::emit_op_mod):
Removed unnecessary #if CPU(X86_64) check to this 32 bit only file.
* offlineasm/x86.rb:
Fixed Windows callee saves naming.
* runtime/VM.cpp:
(JSC::VM::VM):
* runtime/VM.h:
(JSC::VM::calleeSaveRegistersBufferOffset):
(JSC::VM::getAllCalleeSaveRegistersMap):
Provide a RegisterSaveMap that has all registers that might be saved. Added a callee save buffer to be
used for OSR exit and for exception processing in a future patch.
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@189575 268f45cc-cd09-0410-ab3c-d52691b4dbfc
diff --git a/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp b/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp
index 57900e6..1b19051 100644
--- a/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp
+++ b/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp
@@ -211,7 +211,7 @@
sizeof(EncodedJSValue) * (
exit.m_values.size() + numMaterializations + maxMaterializationNumArguments) +
requiredScratchMemorySizeInBytes() +
- jitCode->unwindInfo.m_registers.size() * sizeof(uint64_t));
+ codeBlock->calleeSaveRegisters()->size() * sizeof(uint64_t));
EncodedJSValue* scratch = scratchBuffer ? static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer()) : 0;
EncodedJSValue* materializationPointers = scratch + exit.m_values.size();
EncodedJSValue* materializationArguments = materializationPointers + numMaterializations;
@@ -384,8 +384,8 @@
// Before we start messing with the frame, we need to set aside any registers that the
// FTL code was preserving.
- for (unsigned i = jitCode->unwindInfo.m_registers.size(); i--;) {
- RegisterAtOffset entry = jitCode->unwindInfo.m_registers[i];
+ for (unsigned i = codeBlock->calleeSaveRegisters()->size(); i--;) {
+ RegisterAtOffset entry = codeBlock->calleeSaveRegisters()->at(i);
jit.load64(
MacroAssembler::Address(MacroAssembler::framePointerRegister, entry.offset()),
GPRInfo::regT0);
@@ -432,10 +432,12 @@
jit.add32(GPRInfo::regT3, GPRInfo::regT2);
arityIntact.link(&jit);
+ CodeBlock* baselineCodeBlock = jit.baselineCodeBlockFor(exit.m_codeOrigin);
+
// First set up SP so that our data doesn't get clobbered by signals.
unsigned conservativeStackDelta =
registerPreservationOffset() +
- exit.m_values.numberOfLocals() * sizeof(Register) +
+ (exit.m_values.numberOfLocals() + baselineCodeBlock->calleeSaveSpaceAsVirtualRegisters()) * sizeof(Register) +
maxFrameExtentForSlowPathCall;
conservativeStackDelta = WTF::roundUpToMultipleOf(
stackAlignmentBytes(), conservativeStackDelta);
@@ -457,67 +459,59 @@
jit.store64(GPRInfo::regT0, GPRInfo::regT1);
jit.addPtr(MacroAssembler::TrustedImm32(sizeof(Register)), GPRInfo::regT1);
jit.branchTest32(MacroAssembler::NonZero, GPRInfo::regT2).linkTo(loop, &jit);
-
- // At this point regT1 points to where we would save our registers. Save them here.
- ptrdiff_t currentOffset = 0;
- for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) {
- if (!toSave.get(reg))
- continue;
- currentOffset += sizeof(Register);
- unsigned unwindIndex = jitCode->unwindInfo.indexOf(reg);
- if (unwindIndex == UINT_MAX) {
- // The FTL compilation didn't preserve this register. This means that it also
- // didn't use the register. So its value at the beginning of OSR exit should be
- // preserved by the thunk. Luckily, we saved all registers into the register
- // scratch buffer, so we can restore them from there.
- jit.load64(registerScratch + offsetOfReg(reg), GPRInfo::regT0);
- } else {
- // The FTL compilation preserved the register. Its new value is therefore
- // irrelevant, but we can get the value that was preserved by using the unwind
- // data. We've already copied all unwind-able preserved registers into the unwind
- // scratch buffer, so we can get it from there.
- jit.load64(unwindScratch + unwindIndex, GPRInfo::regT0);
- }
- jit.store64(GPRInfo::regT0, AssemblyHelpers::Address(GPRInfo::regT1, currentOffset));
- }
-
- // We need to make sure that we return into the register restoration thunk. This works
- // differently depending on whether or not we had arity issues.
- MacroAssembler::Jump arityIntactForReturnPC = jit.branch32(
- MacroAssembler::GreaterThanOrEqual,
- CCallHelpers::payloadFor(JSStack::ArgumentCount),
- MacroAssembler::TrustedImm32(codeBlock->numParameters()));
-
- // The return PC in the call frame header points at exactly the right arity restoration
- // thunk. We don't want to change that. But the arity restoration thunk's frame has a
- // return PC and we want to reroute that to our register restoration thunk. The arity
- // restoration's return PC just just below regT1, and the register restoration's return PC
- // is right at regT1.
- jit.loadPtr(MacroAssembler::Address(GPRInfo::regT1, -static_cast<ptrdiff_t>(sizeof(Register))), GPRInfo::regT0);
- jit.storePtr(GPRInfo::regT0, GPRInfo::regT1);
- jit.storePtr(
- MacroAssembler::TrustedImmPtr(vm->getCTIStub(registerRestorationThunkGenerator).code().executableAddress()),
- MacroAssembler::Address(GPRInfo::regT1, -static_cast<ptrdiff_t>(sizeof(Register))));
-
- MacroAssembler::Jump arityReturnPCReady = jit.jump();
- arityIntactForReturnPC.link(&jit);
-
- jit.loadPtr(MacroAssembler::Address(MacroAssembler::framePointerRegister, CallFrame::returnPCOffset()), GPRInfo::regT0);
- jit.storePtr(GPRInfo::regT0, GPRInfo::regT1);
- jit.storePtr(
- MacroAssembler::TrustedImmPtr(vm->getCTIStub(registerRestorationThunkGenerator).code().executableAddress()),
- MacroAssembler::Address(MacroAssembler::framePointerRegister, CallFrame::returnPCOffset()));
-
- arityReturnPCReady.link(&jit);
-
+ RegisterAtOffsetList* baselineCalleeSaves = baselineCodeBlock->calleeSaveRegisters();
+
+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) {
+ if (!toSave.get(reg) || !reg.isGPR())
+ continue;
+ unsigned unwindIndex = codeBlock->calleeSaveRegisters()->indexOf(reg);
+ RegisterAtOffset* baselineRegisterOffset = baselineCalleeSaves->find(reg);
+
+ if (reg.isGPR()) {
+ GPRReg regToLoad = baselineRegisterOffset ? GPRInfo::regT0 : reg.gpr();
+
+ if (unwindIndex == UINT_MAX) {
+ // The FTL compilation didn't preserve this register. This means that it also
+ // didn't use the register. So its value at the beginning of OSR exit should be
+ // preserved by the thunk. Luckily, we saved all registers into the register
+ // scratch buffer, so we can restore them from there.
+ jit.load64(registerScratch + offsetOfReg(reg), regToLoad);
+ } else {
+ // The FTL compilation preserved the register. Its new value is therefore
+ // irrelevant, but we can get the value that was preserved by using the unwind
+ // data. We've already copied all unwind-able preserved registers into the unwind
+ // scratch buffer, so we can get it from there.
+ jit.load64(unwindScratch + unwindIndex, regToLoad);
+ }
+
+ if (baselineRegisterOffset)
+ jit.store64(regToLoad, MacroAssembler::Address(MacroAssembler::framePointerRegister, baselineRegisterOffset->offset()));
+ } else {
+ FPRReg fpRegToLoad = baselineRegisterOffset ? FPRInfo::fpRegT0 : reg.fpr();
+
+ if (unwindIndex == UINT_MAX)
+ jit.loadDouble(MacroAssembler::TrustedImmPtr(registerScratch + offsetOfReg(reg)), fpRegToLoad);
+ else
+ jit.loadDouble(MacroAssembler::TrustedImmPtr(unwindScratch + unwindIndex), fpRegToLoad);
+
+ if (baselineRegisterOffset)
+ jit.storeDouble(fpRegToLoad, MacroAssembler::Address(MacroAssembler::framePointerRegister, baselineRegisterOffset->offset()));
+ }
+ }
+
+ size_t baselineVirtualRegistersForCalleeSaves = baselineCodeBlock->calleeSaveSpaceAsVirtualRegisters();
+
// Now get state out of the scratch buffer and place it back into the stack. The values are
// already reboxed so we just move them.
for (unsigned index = exit.m_values.size(); index--;) {
- int operand = exit.m_values.operandForIndex(index);
-
+ VirtualRegister reg = exit.m_values.virtualRegisterForIndex(index);
+
+ if (reg.isLocal() && reg.toLocal() < static_cast<int>(baselineVirtualRegistersForCalleeSaves))
+ continue;
+
jit.load64(scratch + index, GPRInfo::regT0);
- jit.store64(GPRInfo::regT0, AssemblyHelpers::addressFor(static_cast<VirtualRegister>(operand)));
+ jit.store64(GPRInfo::regT0, AssemblyHelpers::addressFor(reg));
}
handleExitCounts(jit, exit);
