FTL B3 should support OSR exit
https://bugs.webkit.org/show_bug.cgi?id=151710
Reviewed by Saam Barati.
Source/JavaScriptCore:
This adds OSR exit support using the same style that I established with lazy slow paths. All of
the work is driven by FTL::LowerDFGToLLVM, and from there any work that needs to be deferred
until after B3 finishes is attached to the stackmap generator. In order to make it easy to port
all of the different forms of OSR exit - invalidation points, exceptions, etc. - the logic for
registering an OSR exit is abstracted behind OSRExitDescriptor and OSRExitHandle.
An issue that I encountered repeatedly in this patch is OSRExitDescriptor being passed as a
reference (&) rather than pointer (*). The new code uses a lot of lambdas that run after the
current frame pops, so the capture list cannot be [&]. I believe that always listing all of the
captured variables is not scalable considering how sophisticated our use of lambdas is. So, it
makes sense to use [=]. But anytime we captured a variable whose type was OSRExitDescriptor&, it
would be captured by value, because that's how references work. One has to be mindful of these
things whenever using [=]. Note that it's not enough to say that we should have listed the
captured variables explicitly - in that case, we still could have made the mistake by forgetting
to put & in front of the variant. The pattern that worked for me to reason about whether I'm
capturing an object or a pointer to an object is to always use pointer types for pointers: either
RefPtr<> when we also want the lambda to prolong the object's life, or * if we are confident that
the object will stay alive. For this reason, this patch changes all code that references
OSRExitDescriptor to use * instead of &. Consistency makes the code easier to grok, and it made
it easier to introduce the required uses of * in places where there were lambdas.
I tested this by running imaging-gaussian-blur, and running some tests that reqiure OSR exit. I'm
not promising that all kinds of exits work, but we have to begin somewhere.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3Compilation.cpp:
(JSC::B3::Compilation::Compilation):
(JSC::B3::Compilation::~Compilation):
* b3/B3Procedure.cpp:
(JSC::B3::Procedure::addDataSection):
(JSC::B3::Procedure::frameSize):
(JSC::B3::Procedure::calleeSaveRegisters):
* b3/B3Procedure.h:
(JSC::B3::Procedure::releaseByproducts):
(JSC::B3::Procedure::code):
(JSC::B3::Procedure::takeByproducts): Deleted.
* b3/air/AirCode.h:
(JSC::B3::Air::Code::setFrameSize):
(JSC::B3::Air::Code::calleeSaveRegisters):
* b3/air/AirGenerationContext.h:
* ftl/FTLB3Compile.cpp:
(JSC::FTL::compile):
* ftl/FTLCompile.cpp:
(JSC::FTL::mmAllocateDataSection):
* ftl/FTLExceptionHandlerManager.cpp:
(JSC::FTL::ExceptionHandlerManager::lazySlowPathExceptionTarget):
(JSC::FTL::ExceptionHandlerManager::getCallOSRExitCommon):
* ftl/FTLExitThunkGenerator.cpp:
* ftl/FTLExitThunkGenerator.h:
* ftl/FTLJITCode.cpp:
(JSC::FTL::JITCode::JITCode):
(JSC::FTL::JITCode::initializeB3Code):
(JSC::FTL::JITCode::initializeB3Byproducts):
(JSC::FTL::JITCode::initializeExitThunks):
(JSC::FTL::JITCode::validateReferences):
(JSC::FTL::JITCode::liveRegistersToPreserveAtExceptionHandlingCallSite):
* ftl/FTLJITCode.h:
* ftl/FTLJITFinalizer.cpp:
(JSC::FTL::JITFinalizer::finalizeFunction):
* ftl/FTLJITFinalizer.h:
* ftl/FTLJSCall.cpp:
(JSC::FTL::JSCall::emit):
* ftl/FTLJSCallBase.cpp:
(JSC::FTL::JSCallBase::emit):
* ftl/FTLJSTailCall.cpp:
(JSC::FTL::JSTailCall::JSTailCall):
(JSC::FTL::JSTailCall::emit):
(JSC::FTL::DFG::getRegisterWithAddend): Deleted.
(JSC::FTL::m_instructionOffset): Deleted.
* ftl/FTLJSTailCall.h:
(JSC::FTL::JSTailCall::patchpoint):
(JSC::FTL::JSTailCall::stackmapID):
(JSC::FTL::JSTailCall::estimatedSize):
(JSC::FTL::JSTailCall::operator<):
* ftl/FTLLowerDFGToLLVM.cpp:
(JSC::FTL::DFG::LowerDFGToLLVM::compileInvalidationPoint):
(JSC::FTL::DFG::LowerDFGToLLVM::appendOSRExitArgumentsForPatchpointIfWillCatchException):
(JSC::FTL::DFG::LowerDFGToLLVM::lowBlock):
(JSC::FTL::DFG::LowerDFGToLLVM::appendOSRExitDescriptor):
(JSC::FTL::DFG::LowerDFGToLLVM::appendOSRExit):
(JSC::FTL::DFG::LowerDFGToLLVM::blessSpeculation):
(JSC::FTL::DFG::LowerDFGToLLVM::emitOSRExitCall):
(JSC::FTL::DFG::LowerDFGToLLVM::buildExitArguments):
(JSC::FTL::DFG::LowerDFGToLLVM::callStackmap):
(JSC::FTL::lowerDFGToLLVM):
* ftl/FTLOSRExit.cpp:
(JSC::FTL::OSRExitDescriptor::OSRExitDescriptor):
(JSC::FTL::OSRExitDescriptor::validateReferences):
(JSC::FTL::OSRExitDescriptor::appendOSRExit):
(JSC::FTL::OSRExitDescriptor::appendOSRExitLater):
(JSC::FTL::OSRExitDescriptor::prepareOSRExitHandle):
(JSC::FTL::OSRExit::OSRExit):
(JSC::FTL::OSRExit::codeLocationForRepatch):
(JSC::FTL::OSRExit::gatherRegistersToSpillForCallIfException):
(JSC::FTL::OSRExit::spillRegistersToSpillSlot):
(JSC::FTL::OSRExit::recoverRegistersFromSpillSlot):
(JSC::FTL::OSRExit::willArriveAtExitFromIndirectExceptionCheck):
* ftl/FTLOSRExit.h:
(JSC::FTL::OSRExit::considerAddingAsFrequentExitSite):
* ftl/FTLOSRExitCompilationInfo.h:
(JSC::FTL::OSRExitCompilationInfo::OSRExitCompilationInfo):
* ftl/FTLOSRExitCompiler.cpp:
(JSC::FTL::reboxAccordingToFormat):
(JSC::FTL::compileRecovery):
(JSC::FTL::compileStub):
(JSC::FTL::compileFTLOSRExit):
* ftl/FTLOSRExitHandle.cpp: Added.
(JSC::FTL::OSRExitHandle::emitExitThunk):
* ftl/FTLOSRExitHandle.h: Added.
(JSC::FTL::OSRExitHandle::OSRExitHandle):
* ftl/FTLState.cpp:
(JSC::FTL::State::State):
(JSC::FTL::State::~State):
Source/WTF:
Make sure that this has perfect forwarding.
* wtf/SegmentedVector.h:
(WTF::SegmentedVector::append):
(WTF::SegmentedVector::alloc):
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@193362 268f45cc-cd09-0410-ab3c-d52691b4dbfc
diff --git a/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp b/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp
index a9a707d..f4edfed 100644
--- a/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp
+++ b/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp
@@ -98,7 +98,12 @@
}
static void compileRecovery(
- CCallHelpers& jit, const ExitValue& value, StackMaps::Record* record, StackMaps& stackmaps,
+ CCallHelpers& jit, const ExitValue& value,
+#if FTL_USES_B3
+ Vector<B3::ValueRep>& valueReps,
+#else // FTL_USES_B3
+ StackMaps::Record* record, StackMaps& stackmaps,
+#endif // FTL_USES_B3
char* registerScratch,
const HashMap<ExitTimeObjectMaterialization*, EncodedJSValue*>& materializationToPointer)
{
@@ -112,8 +117,13 @@
break;
case ExitValueArgument:
+#if FTL_USES_B3
+ Location::forValueRep(valueReps[value.exitArgument().argument()]).restoreInto(
+ jit, registerScratch, GPRInfo::regT0);
+#else // FTL_USES_B3
record->locations[value.exitArgument().argument()].restoreInto(
jit, stackmaps, registerScratch, GPRInfo::regT0);
+#endif // FTL_USES_B3
break;
case ExitValueInJSStack:
@@ -124,10 +134,17 @@
break;
case ExitValueRecovery:
+#if FTL_USES_B3
+ Location::forValueRep(valueReps[value.rightRecoveryArgument()]).restoreInto(
+ jit, registerScratch, GPRInfo::regT1);
+ Location::forValueRep(valueReps[value.leftRecoveryArgument()]).restoreInto(
+ jit, registerScratch, GPRInfo::regT0);
+#else // FTL_USES_B3
record->locations[value.rightRecoveryArgument()].restoreInto(
jit, stackmaps, registerScratch, GPRInfo::regT1);
record->locations[value.leftRecoveryArgument()].restoreInto(
jit, stackmaps, registerScratch, GPRInfo::regT0);
+#endif // FTL_USES_B3
switch (value.recoveryOpcode()) {
case AddRecovery:
switch (value.recoveryFormat()) {
@@ -177,8 +194,12 @@
static void compileStub(
unsigned exitID, JITCode* jitCode, OSRExit& exit, VM* vm, CodeBlock* codeBlock)
{
+#if FTL_USES_B3
+ UNUSED_PARAM(jitCode);
+#else // FTL_USES_B3
StackMaps::Record* record = &jitCode->stackmaps.records[exit.m_stackmapRecordIndex];
- RELEASE_ASSERT(record->patchpointID == exit.m_descriptor.m_stackmapID);
+ RELEASE_ASSERT(record->patchpointID == exit.m_descriptor->m_stackmapID);
+#endif // FTL_USES_B3
// This code requires framePointerRegister is the same as callFrameRegister
static_assert(MacroAssembler::framePointerRegister == GPRInfo::callFrameRegister, "MacroAssembler::framePointerRegister and GPRInfo::callFrameRegister must be the same");
@@ -192,7 +213,7 @@
// Figure out how much space we need for those object allocations.
unsigned numMaterializations = 0;
size_t maxMaterializationNumArguments = 0;
- for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor.m_materializations) {
+ for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor->m_materializations) {
numMaterializations++;
maxMaterializationNumArguments = std::max(
@@ -202,21 +223,32 @@
ScratchBuffer* scratchBuffer = vm->scratchBufferForSize(
sizeof(EncodedJSValue) * (
- exit.m_descriptor.m_values.size() + numMaterializations + maxMaterializationNumArguments) +
+ exit.m_descriptor->m_values.size() + numMaterializations + maxMaterializationNumArguments) +
requiredScratchMemorySizeInBytes() +
codeBlock->calleeSaveRegisters()->size() * sizeof(uint64_t));
EncodedJSValue* scratch = scratchBuffer ? static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer()) : 0;
- EncodedJSValue* materializationPointers = scratch + exit.m_descriptor.m_values.size();
+ EncodedJSValue* materializationPointers = scratch + exit.m_descriptor->m_values.size();
EncodedJSValue* materializationArguments = materializationPointers + numMaterializations;
char* registerScratch = bitwise_cast<char*>(materializationArguments + maxMaterializationNumArguments);
uint64_t* unwindScratch = bitwise_cast<uint64_t*>(registerScratch + requiredScratchMemorySizeInBytes());
HashMap<ExitTimeObjectMaterialization*, EncodedJSValue*> materializationToPointer;
unsigned materializationCount = 0;
- for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor.m_materializations) {
+ for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor->m_materializations) {
materializationToPointer.add(
materialization, materializationPointers + materializationCount++);
}
+
+ auto recoverValue = [&] (const ExitValue& value) {
+ compileRecovery(
+ jit, value,
+#if FTL_USES_B3
+ exit.m_valueReps,
+#else // FTL_USES_B3
+ record, jitCode->stackmaps,
+#endif // FTL_USES_B3
+ registerScratch, materializationToPointer);
+ };
// Note that we come in here, the stack used to be as LLVM left it except that someone called pushToSave().
// We don't care about the value they saved. But, we do appreciate the fact that they did it, because we use
@@ -247,10 +279,14 @@
jit.move(MacroAssembler::TrustedImm64(TagMask), GPRInfo::tagMaskRegister);
// Do some value profiling.
- if (exit.m_descriptor.m_profileDataFormat != DataFormatNone) {
+ if (exit.m_descriptor->m_profileDataFormat != DataFormatNone) {
+#if FTL_USES_B3
+ Location::forValueRep(exit.m_valueReps[0]).restoreInto(jit, registerScratch, GPRInfo::regT0);
+#else // FTL_USES_B3
record->locations[0].restoreInto(jit, jitCode->stackmaps, registerScratch, GPRInfo::regT0);
+#endif // FTL_USES_B3
reboxAccordingToFormat(
- exit.m_descriptor.m_profileDataFormat, jit, GPRInfo::regT0, GPRInfo::regT1, GPRInfo::regT2);
+ exit.m_descriptor->m_profileDataFormat, jit, GPRInfo::regT0, GPRInfo::regT1, GPRInfo::regT2);
if (exit.m_kind == BadCache || exit.m_kind == BadIndexingType) {
CodeOrigin codeOrigin = exit.m_codeOriginForExitProfile;
@@ -264,8 +300,8 @@
}
}
- if (!!exit.m_descriptor.m_valueProfile)
- jit.store64(GPRInfo::regT0, exit.m_descriptor.m_valueProfile.getSpecFailBucket(0));
+ if (!!exit.m_descriptor->m_valueProfile)
+ jit.store64(GPRInfo::regT0, exit.m_descriptor->m_valueProfile.getSpecFailBucket(0));
}
// Materialize all objects. Don't materialize an object until all
@@ -275,7 +311,7 @@
// allocation of the former.
HashSet<ExitTimeObjectMaterialization*> toMaterialize;
- for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor.m_materializations)
+ for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor->m_materializations)
toMaterialize.add(materialization);
while (!toMaterialize.isEmpty()) {
@@ -307,13 +343,10 @@
// We only recover the fields that are needed for the allocation.
for (unsigned propertyIndex = materialization->properties().size(); propertyIndex--;) {
const ExitPropertyValue& property = materialization->properties()[propertyIndex];
- const ExitValue& value = property.value();
if (!property.location().neededForMaterialization())
continue;
- compileRecovery(
- jit, value, record, jitCode->stackmaps, registerScratch,
- materializationToPointer);
+ recoverValue(property.value());
jit.storePtr(GPRInfo::regT0, materializationArguments + propertyIndex);
}
@@ -338,12 +371,9 @@
// Now that all the objects have been allocated, we populate them
// with the correct values. This time we can recover all the
// fields, including those that are only needed for the allocation.
- for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor.m_materializations) {
+ for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor->m_materializations) {
for (unsigned propertyIndex = materialization->properties().size(); propertyIndex--;) {
- const ExitValue& value = materialization->properties()[propertyIndex].value();
- compileRecovery(
- jit, value, record, jitCode->stackmaps, registerScratch,
- materializationToPointer);
+ recoverValue(materialization->properties()[propertyIndex].value());
jit.storePtr(GPRInfo::regT0, materializationArguments + propertyIndex);
}
@@ -359,16 +389,14 @@
// Save all state from wherever the exit data tells us it was, into the appropriate place in
// the scratch buffer. This also does the reboxing.
- for (unsigned index = exit.m_descriptor.m_values.size(); index--;) {
- compileRecovery(
- jit, exit.m_descriptor.m_values[index], record, jitCode->stackmaps, registerScratch,
- materializationToPointer);
+ for (unsigned index = exit.m_descriptor->m_values.size(); index--;) {
+ recoverValue(exit.m_descriptor->m_values[index]);
jit.store64(GPRInfo::regT0, scratch + index);
}
// Henceforth we make it look like the exiting function was called through a register
// preservation wrapper. This implies that FP must be nudged down by a certain amount. Then
- // we restore the various things according to either exit.m_descriptor.m_values or by copying from the
+ // we restore the various things according to either exit.m_descriptor->m_values or by copying from the
// old frame, and finally we save the various callee-save registers into where the
// restoration thunk would restore them from.
@@ -416,7 +444,7 @@
// First set up SP so that our data doesn't get clobbered by signals.
unsigned conservativeStackDelta =
- (exit.m_descriptor.m_values.numberOfLocals() + baselineCodeBlock->calleeSaveSpaceAsVirtualRegisters()) * sizeof(Register) +
+ (exit.m_descriptor->m_values.numberOfLocals() + baselineCodeBlock->calleeSaveSpaceAsVirtualRegisters()) * sizeof(Register) +
maxFrameExtentForSlowPathCall;
conservativeStackDelta = WTF::roundUpToMultipleOf(
stackAlignmentBytes(), conservativeStackDelta);
@@ -493,8 +521,8 @@
// Now get state out of the scratch buffer and place it back into the stack. The values are
// already reboxed so we just move them.
- for (unsigned index = exit.m_descriptor.m_values.size(); index--;) {
- VirtualRegister reg = exit.m_descriptor.m_values.virtualRegisterForIndex(index);
+ for (unsigned index = exit.m_descriptor->m_values.size(); index--;) {
+ VirtualRegister reg = exit.m_descriptor->m_values.virtualRegisterForIndex(index);
if (reg.isLocal() && reg.toLocal() < static_cast<int>(baselineVirtualRegistersForCalleeSaves))
continue;
@@ -511,11 +539,19 @@
exit.m_code = FINALIZE_CODE_IF(
shouldDumpDisassembly() || Options::verboseOSR() || Options::verboseFTLOSRExit(),
patchBuffer,
+#if FTL_USES_B3
+ ("FTL OSR exit #%u (%s, %s) from %s, with operands = %s",
+ exitID, toCString(exit.m_codeOrigin).data(),
+ exitKindToString(exit.m_kind), toCString(*codeBlock).data(),
+ toCString(ignoringContext<DumpContext>(exit.m_descriptor->m_values)).data())
+#else // FTL_USES_B3
("FTL OSR exit #%u (%s, %s) from %s, with operands = %s, and record = %s",
exitID, toCString(exit.m_codeOrigin).data(),
exitKindToString(exit.m_kind), toCString(*codeBlock).data(),
- toCString(ignoringContext<DumpContext>(exit.m_descriptor.m_values)).data(),
- toCString(*record).data()));
+ toCString(ignoringContext<DumpContext>(exit.m_descriptor->m_values)).data(),
+ toCString(*record).data())
+#endif // FTL_USES_B3
+ );
}
extern "C" void* compileFTLOSRExit(ExecState* exec, unsigned exitID)
@@ -544,15 +580,15 @@
dataLog(" Origin: ", exit.m_codeOrigin, "\n");
if (exit.m_codeOriginForExitProfile != exit.m_codeOrigin)
dataLog(" Origin for exit profile: ", exit.m_codeOriginForExitProfile, "\n");
- dataLog(" Exit stackmap ID: ", exit.m_descriptor.m_stackmapID, "\n");
+ dataLog(" Exit stackmap ID: ", exit.m_descriptor->m_stackmapID, "\n");
dataLog(" Current call site index: ", exec->callSiteIndex().bits(), "\n");
dataLog(" Exit is exception handler: ", exit.m_isExceptionHandler,
" will arrive at exit from genericUnwind(): ", exit.willArriveAtOSRExitFromGenericUnwind(),
- " will arrive at exit from lazy slow path: ", exit.m_descriptor.m_exceptionType == ExceptionType::LazySlowPath, "\n");
- dataLog(" Exit values: ", exit.m_descriptor.m_values, "\n");
- if (!exit.m_descriptor.m_materializations.isEmpty()) {
+ " will arrive at exit from lazy slow path: ", exit.m_descriptor->m_exceptionType == ExceptionType::LazySlowPath, "\n");
+ dataLog(" Exit values: ", exit.m_descriptor->m_values, "\n");
+ if (!exit.m_descriptor->m_materializations.isEmpty()) {
dataLog(" Materializations:\n");
- for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor.m_materializations)
+ for (ExitTimeObjectMaterialization* materialization : exit.m_descriptor->m_materializations)
dataLog(" ", pointerDump(materialization), "\n");
}
}
