GC should support isoheaps
https://bugs.webkit.org/show_bug.cgi?id=179288
Reviewed by Saam Barati.
Source/JavaScriptCore:
This expands the power of the Subspace API in JSC:
- Everything associated with describing the types of objects is now part of the HeapCellType class.
We have different HeapCellTypes for different destruction strategies. Any Subspace can use any
HeapCellType; these are orthogonal things.
- There are now two variants of Subspace: CompleteSubspace, which can allocate any size objects using
any AlignedMemoryAllocator; and IsoSubspace, which can allocate just one size of object and uses a
special virtual memory pool for that purpose. Like bmalloc's IsoHeap, IsoSubspace hoards virtual
pages but releases the physical pages as part of the respective allocator's scavenging policy
(the Scavenger in bmalloc for IsoHeap and the incremental sweep and full sweep in Riptide for
IsoSubspace).
So far, this patch just puts subtypes of ExecutableBase in IsoSubspaces. If it works, we can use it
for more things.
This does not have any effect on JetStream (0.18% faster with p = 0.69).
* JavaScriptCore.xcodeproj/project.pbxproj:
* Sources.txt:
* bytecode/AccessCase.cpp:
(JSC::AccessCase::generateImpl):
* bytecode/ObjectAllocationProfileInlines.h:
(JSC::ObjectAllocationProfile::initializeProfile):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
* heap/AlignedMemoryAllocator.cpp:
(JSC::AlignedMemoryAllocator::registerAllocator):
(JSC::AlignedMemoryAllocator::registerSubspace):
* heap/AlignedMemoryAllocator.h:
(JSC::AlignedMemoryAllocator::firstAllocator const):
* heap/AllocationFailureMode.h: Added.
* heap/CompleteSubspace.cpp: Added.
(JSC::CompleteSubspace::CompleteSubspace):
(JSC::CompleteSubspace::~CompleteSubspace):
(JSC::CompleteSubspace::allocatorFor):
(JSC::CompleteSubspace::allocate):
(JSC::CompleteSubspace::allocateNonVirtual):
(JSC::CompleteSubspace::allocatorForSlow):
(JSC::CompleteSubspace::allocateSlow):
(JSC::CompleteSubspace::tryAllocateSlow):
* heap/CompleteSubspace.h: Added.
(JSC::CompleteSubspace::offsetOfAllocatorForSizeStep):
(JSC::CompleteSubspace::allocatorForSizeStep):
(JSC::CompleteSubspace::allocatorForNonVirtual):
* heap/HeapCellType.cpp: Added.
(JSC::HeapCellType::HeapCellType):
(JSC::HeapCellType::~HeapCellType):
(JSC::HeapCellType::finishSweep):
(JSC::HeapCellType::destroy):
* heap/HeapCellType.h: Added.
(JSC::HeapCellType::attributes const):
* heap/IsoAlignedMemoryAllocator.cpp: Added.
(JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator):
(JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator):
(JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory):
(JSC::IsoAlignedMemoryAllocator::freeAlignedMemory):
(JSC::IsoAlignedMemoryAllocator::dump const):
* heap/IsoAlignedMemoryAllocator.h: Added.
* heap/IsoSubspace.cpp: Added.
(JSC::IsoSubspace::IsoSubspace):
(JSC::IsoSubspace::~IsoSubspace):
(JSC::IsoSubspace::allocatorFor):
(JSC::IsoSubspace::allocatorForNonVirtual):
(JSC::IsoSubspace::allocate):
(JSC::IsoSubspace::allocateNonVirtual):
* heap/IsoSubspace.h: Added.
(JSC::IsoSubspace::size const):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::setSubspace):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase): Deleted.
(JSC::MarkedAllocator::allocateSlowCaseImpl): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::nextAllocatorInAlignedMemoryAllocator const):
(JSC::MarkedAllocator::setNextAllocatorInAlignedMemoryAllocator):
* heap/MarkedAllocatorInlines.h:
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::tryAllocate): Deleted.
* heap/MarkedBlock.h:
* heap/MarkedBlockInlines.h:
(JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType):
(JSC::MarkedBlock::Handle::finishSweepKnowingSubspace): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::addMarkedAllocator):
* heap/MarkedSpace.h:
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::initialize):
(JSC::Subspace::finishSweep):
(JSC::Subspace::destroy):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
(): Deleted.
(JSC::Subspace::allocate): Deleted.
(JSC::Subspace::tryAllocate): Deleted.
(JSC::Subspace::allocatorForSlow): Deleted.
(JSC::Subspace::allocateSlow): Deleted.
(JSC::Subspace::tryAllocateSlow): Deleted.
(JSC::Subspace::didAllocate): Deleted.
* heap/Subspace.h:
(JSC::Subspace::heapCellType const):
(JSC::Subspace::nextSubspaceInAlignedMemoryAllocator const):
(JSC::Subspace::setNextSubspaceInAlignedMemoryAllocator):
(JSC::Subspace::offsetOfAllocatorForSizeStep): Deleted.
(JSC::Subspace::allocatorForSizeStep): Deleted.
(JSC::Subspace::tryAllocatorFor): Deleted.
(JSC::Subspace::allocatorFor): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
(JSC::AssemblyHelpers::emitAllocateVariableSizedCell):
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
* runtime/ButterflyInlines.h:
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::tryCreate):
(JSC::Butterfly::growArrayRight):
* runtime/DirectArguments.cpp:
(JSC::DirectArguments::overrideThings):
* runtime/DirectArguments.h:
(JSC::DirectArguments::subspaceFor):
* runtime/DirectEvalExecutable.h:
* runtime/EvalExecutable.h:
* runtime/ExecutableBase.h:
(JSC::ExecutableBase::subspaceFor):
* runtime/FunctionExecutable.h:
* runtime/GenericArgumentsInlines.h:
(JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
* runtime/HashMapImpl.h:
(JSC::HashMapBuffer::create):
* runtime/IndirectEvalExecutable.h:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitializedRestricted):
(JSC::JSArray::unshiftCountSlowCase):
* runtime/JSArray.h:
(JSC::JSArray::tryCreate):
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
* runtime/JSCell.h:
(JSC::subspaceFor):
* runtime/JSCellInlines.h:
(JSC::JSCell::subspaceFor):
(JSC::tryAllocateCellHelper):
(JSC::allocateCell):
(JSC::tryAllocateCell):
* runtime/JSDestructibleObject.h:
(JSC::JSDestructibleObject::subspaceFor):
* runtime/JSDestructibleObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.cpp.
(JSC::JSDestructibleObjectHeapCellType::JSDestructibleObjectHeapCellType):
(JSC::JSDestructibleObjectHeapCellType::~JSDestructibleObjectHeapCellType):
(JSC::JSDestructibleObjectHeapCellType::finishSweep):
(JSC::JSDestructibleObjectHeapCellType::destroy):
(JSC::JSDestructibleObjectSubspace::JSDestructibleObjectSubspace): Deleted.
(JSC::JSDestructibleObjectSubspace::~JSDestructibleObjectSubspace): Deleted.
(JSC::JSDestructibleObjectSubspace::finishSweep): Deleted.
(JSC::JSDestructibleObjectSubspace::destroy): Deleted.
* runtime/JSDestructibleObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.h.
* runtime/JSDestructibleObjectSubspace.cpp: Removed.
* runtime/JSDestructibleObjectSubspace.h: Removed.
* runtime/JSLexicalEnvironment.h:
(JSC::JSLexicalEnvironment::subspaceFor):
* runtime/JSSegmentedVariableObject.h:
(JSC::JSSegmentedVariableObject::subspaceFor):
* runtime/JSSegmentedVariableObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.cpp.
(JSC::JSSegmentedVariableObjectHeapCellType::JSSegmentedVariableObjectHeapCellType):
(JSC::JSSegmentedVariableObjectHeapCellType::~JSSegmentedVariableObjectHeapCellType):
(JSC::JSSegmentedVariableObjectHeapCellType::finishSweep):
(JSC::JSSegmentedVariableObjectHeapCellType::destroy):
(JSC::JSSegmentedVariableObjectSubspace::JSSegmentedVariableObjectSubspace): Deleted.
(JSC::JSSegmentedVariableObjectSubspace::~JSSegmentedVariableObjectSubspace): Deleted.
(JSC::JSSegmentedVariableObjectSubspace::finishSweep): Deleted.
(JSC::JSSegmentedVariableObjectSubspace::destroy): Deleted.
* runtime/JSSegmentedVariableObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.h.
* runtime/JSSegmentedVariableObjectSubspace.cpp: Removed.
* runtime/JSSegmentedVariableObjectSubspace.h: Removed.
* runtime/JSString.h:
(JSC::JSString::subspaceFor):
* runtime/JSStringHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.cpp.
(JSC::JSStringHeapCellType::JSStringHeapCellType):
(JSC::JSStringHeapCellType::~JSStringHeapCellType):
(JSC::JSStringHeapCellType::finishSweep):
(JSC::JSStringHeapCellType::destroy):
(JSC::JSStringSubspace::JSStringSubspace): Deleted.
(JSC::JSStringSubspace::~JSStringSubspace): Deleted.
(JSC::JSStringSubspace::finishSweep): Deleted.
(JSC::JSStringSubspace::destroy): Deleted.
* runtime/JSStringHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.h.
* runtime/JSStringSubspace.cpp: Removed.
* runtime/JSStringSubspace.h: Removed.
* runtime/ModuleProgramExecutable.h:
* runtime/NativeExecutable.h:
* runtime/ProgramExecutable.h:
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
* runtime/ScopedArguments.h:
(JSC::ScopedArguments::subspaceFor):
* runtime/VM.cpp:
(JSC::VM::VM):
* runtime/VM.h:
(JSC::VM::gigacageAuxiliarySpace):
* wasm/js/JSWebAssemblyCodeBlock.h:
* wasm/js/JSWebAssemblyCodeBlockHeapCellType.cpp: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.cpp.
(JSC::JSWebAssemblyCodeBlockHeapCellType::JSWebAssemblyCodeBlockHeapCellType):
(JSC::JSWebAssemblyCodeBlockHeapCellType::~JSWebAssemblyCodeBlockHeapCellType):
(JSC::JSWebAssemblyCodeBlockHeapCellType::finishSweep):
(JSC::JSWebAssemblyCodeBlockHeapCellType::destroy):
(JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): Deleted.
(JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): Deleted.
(JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): Deleted.
(JSC::JSWebAssemblyCodeBlockSubspace::destroy): Deleted.
* wasm/js/JSWebAssemblyCodeBlockHeapCellType.h: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.h.
* wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Removed.
* wasm/js/JSWebAssemblyCodeBlockSubspace.h: Removed.
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no new behavior.
Adopting changes in JSC Subspace API.
* ForwardingHeaders/runtime/JSDestructibleObjectHeapCellType.h: Added.
* ForwardingHeaders/runtime/JSSegmentedVariableObjectHeapCellType.h: Added.
* bindings/js/JSDOMWrapper.cpp:
(WebCore::outputConstraintSubspaceFor):
(WebCore::globalObjectOutputConstraintSubspaceFor):
* bindings/js/JSDOMWrapper.h:
* bindings/js/WebCoreJSClientData.cpp:
(WebCore::JSVMClientData::JSVMClientData):
* bindings/js/WebCoreJSClientData.h:
(WebCore::JSVMClientData::outputConstraintSpace):
(WebCore::JSVMClientData::globalObjectOutputConstraintSpace):
* bindings/scripts/CodeGeneratorJS.pm:
(GenerateHeader):
Source/WTF:
One of my favorite data structures in the GC is a singly-linked list that knows its tail, so that
things get added to it at the end rather that at the beginning. In this patch, I use this to put
the same node on multiple lists, which our existing linked list templates also don't support.
This adds a new linked list that does those things:
- It supports append(). It could also support prepend(), but currently there is no need for it.
- It supports nodes that are on multiple lists. The GC uses std::mem_fn() to create a lambda that the
list uses to set next.
* WTF.xcodeproj/project.pbxproj:
* wtf/SinglyLinkedListWithTail.h: Added.
(WTF::SinglyLinkedListWithTail::isEmpty const):
(WTF::SinglyLinkedListWithTail::append):
(WTF::SinglyLinkedListWithTail::first const):
(WTF::SinglyLinkedListWithTail::last const):
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@225314 268f45cc-cd09-0410-ab3c-d52691b4dbfc
diff --git a/Source/JavaScriptCore/runtime/JSArray.h b/Source/JavaScriptCore/runtime/JSArray.h
index abf4ea1..9ea3977 100644
--- a/Source/JavaScriptCore/runtime/JSArray.h
+++ b/Source/JavaScriptCore/runtime/JSArray.h
@@ -237,7 +237,9 @@
return nullptr;
unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, vectorLengthHint);
- void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
+ void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(
+ Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)),
+ nullptr, AllocationFailureMode::ReturnNull);
if (!temp)
return nullptr;
butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
