Implement a new allocator for backing stores
https://bugs.webkit.org/show_bug.cgi?id=75181
Reviewed by Filip Pizlo.
Source/JavaScriptCore:
We want to move away from using fastMalloc for the backing stores for
some of our objects (e.g. JSArray, JSObject, JSString, etc). These backing
stores have a nice property in that they only have a single owner (i.e. a
single pointer to them at any one time). One way that we can take advantage
of this property is to implement a simple bump allocator/copying collector,
which will run alongside our normal mark/sweep collector, that only needs to
update the single owner pointer rather than having to redirect an arbitrary
number of pointers in from-space to to-space.
This plan can give us a number of benefits. We can beat fastMalloc in terms
of both performance and memory usage, we can track how much memory we're using
far more accurately than our rough estimation now through the use of
reportExtraMemoryCost, and we can allocate arbitrary size objects (as opposed
to being limited to size classes like we have been historically). This is also
another step toward moving away from lazy destruction, which will improve our memory footprint.
We start by creating said allocator and moving the ArrayStorage for JSArray
to use it rather than fastMalloc.
The design of the collector is as follows:
Allocation:
-The collector allocates 64KB chunks from the OS to use for object allocation.
-Each chunk contains an offset, a flag indicating if the block has been pinned,
and a payload, along with next and prev pointers so that they can be put in DoublyLinkedLists.
-Any allocation greater than 64KB gets its own separate oversize block, which
is managed separately from the rest.
-If the allocator receives a request for more than the remaining amount in the
current block, it grabs a fresh block.
-Grabbing a fresh block means grabbing one off of the global free list (which is now
shared between the mark/sweep allocator and the bump allocator) if there is one.
If there isn't a new one we do one of two things: allocate a new block from the OS
if we're not ready for a GC yet, or run a GC and then try again. If we still don't
have enough space after the GC, we allocate a new block from the OS.
Garbage collection:
-At the start of garbage collection during conservative stack scanning, if we encounter
what appears to be a pointer to a bump-allocated block of memory, we pin that block so
that it will not be copied for this round of collection.
-We also pin any oversize blocks that we encounter, which effectively doubles as a
"mark bit" for that block. Any oversize blocks that aren't pinned at the end of copying
are given back to the OS.
-Marking threads are now also responsible for copying bump-allocated objects to newSpace
-Each marking thread has a private 64KB block into which it copies bump-allocated objects that it encounters.
-When that block fills up, the marking thread gives it back to the allocator and requests a new one.
-When all marking has concluded, each thread gives back its copy block, even if it isn't full.
-At the conclusion of copying (which is done by the end of the marking phase), we un-pin
any pinned blocks and give any blocks left in from-space to the global free list.
* CMakeLists.txt:
* GNUmakefile.list.am:
* JavaScriptCore.gypi:
* JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj:
* JavaScriptCore.vcproj/WTF/WTF.vcproj:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Target.pri:
* heap/AllocationSpace.cpp:
(JSC::AllocationSpace::allocateSlowCase):
(JSC::AllocationSpace::allocateBlock):
(JSC::AllocationSpace::freeBlocks):
* heap/AllocationSpace.h:
(JSC::AllocationSpace::waterMark):
* heap/BumpBlock.h: Added.
(JSC::BumpBlock::BumpBlock):
* heap/BumpSpace.cpp: Added.
(JSC::BumpSpace::tryAllocateSlowCase):
* heap/BumpSpace.h: Added.
(JSC::BumpSpace::isInCopyPhase):
(JSC::BumpSpace::totalMemoryAllocated):
(JSC::BumpSpace::totalMemoryUtilized):
* heap/BumpSpaceInlineMethods.h: Added.
(JSC::BumpSpace::BumpSpace):
(JSC::BumpSpace::init):
(JSC::BumpSpace::contains):
(JSC::BumpSpace::pin):
(JSC::BumpSpace::startedCopying):
(JSC::BumpSpace::doneCopying):
(JSC::BumpSpace::doneFillingBlock):
(JSC::BumpSpace::recycleBlock):
(JSC::BumpSpace::getFreshBlock):
(JSC::BumpSpace::borrowBlock):
(JSC::BumpSpace::addNewBlock):
(JSC::BumpSpace::allocateNewBlock):
(JSC::BumpSpace::fitsInBlock):
(JSC::BumpSpace::fitsInCurrentBlock):
(JSC::BumpSpace::tryAllocate):
(JSC::BumpSpace::tryAllocateOversize):
(JSC::BumpSpace::allocateFromBlock):
(JSC::BumpSpace::tryReallocate):
(JSC::BumpSpace::tryReallocateOversize):
(JSC::BumpSpace::isOversize):
(JSC::BumpSpace::isPinned):
(JSC::BumpSpace::oversizeBlockFor):
(JSC::BumpSpace::blockFor):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::add):
* heap/ConservativeRoots.h:
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::blockFreeingThreadMain):
(JSC::Heap::reportExtraMemoryCostSlowCase):
(JSC::Heap::getConservativeRegisterRoots):
(JSC::Heap::markRoots):
(JSC::Heap::collect):
(JSC::Heap::releaseFreeBlocks):
* heap/Heap.h:
(JSC::Heap::waterMark):
(JSC::Heap::highWaterMark):
(JSC::Heap::setHighWaterMark):
(JSC::Heap::tryAllocateStorage):
(JSC::Heap::tryReallocateStorage):
* heap/HeapBlock.h: Added.
(JSC::HeapBlock::HeapBlock):
* heap/MarkStack.cpp:
(JSC::MarkStackThreadSharedData::MarkStackThreadSharedData):
(JSC::SlotVisitor::drain):
(JSC::SlotVisitor::drainFromShared):
(JSC::SlotVisitor::startCopying):
(JSC::SlotVisitor::allocateNewSpace):
(JSC::SlotVisitor::copy):
(JSC::SlotVisitor::copyAndAppend):
(JSC::SlotVisitor::doneCopying):
* heap/MarkStack.h:
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::recycle):
(JSC::MarkedBlock::MarkedBlock):
* heap/MarkedBlock.h:
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::MarkedSpace):
* heap/MarkedSpace.h:
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::SizeClass::resetAllocator):
* heap/SlotVisitor.h:
(JSC::SlotVisitor::SlotVisitor):
* heap/TinyBloomFilter.h:
(JSC::TinyBloomFilter::reset):
* runtime/JSArray.cpp:
(JSC::JSArray::JSArray):
(JSC::JSArray::finishCreation):
(JSC::JSArray::tryFinishCreationUninitialized):
(JSC::JSArray::~JSArray):
(JSC::JSArray::enterSparseMode):
(JSC::JSArray::defineOwnNumericProperty):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::getOwnPropertySlotByIndex):
(JSC::JSArray::getOwnPropertyDescriptor):
(JSC::JSArray::putByIndexBeyondVectorLength):
(JSC::JSArray::deletePropertyByIndex):
(JSC::JSArray::getOwnPropertyNames):
(JSC::JSArray::increaseVectorLength):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::unshiftCount):
(JSC::JSArray::visitChildren):
(JSC::JSArray::sortNumeric):
(JSC::JSArray::sort):
(JSC::JSArray::compactForSorting):
(JSC::JSArray::subclassData):
(JSC::JSArray::setSubclassData):
(JSC::JSArray::checkConsistency):
* runtime/JSArray.h:
(JSC::JSArray::inSparseMode):
(JSC::JSArray::isLengthWritable):
* wtf/CheckedBoolean.h: Added.
(CheckedBoolean::CheckedBoolean):
(CheckedBoolean::~CheckedBoolean):
(CheckedBoolean::operator bool):
* wtf/DoublyLinkedList.h:
(WTF::::push):
* wtf/StdLibExtras.h:
(WTF::isPointerAligned):
Source/JavaScriptGlue:
Added forwarding header for new CheckedBoolean used in the bump allocator.
* ForwardingHeaders/wtf/CheckedBoolean.h: Added.
Source/WebCore:
No new tests.
Added forwarding header for new CheckedBoolean used in the bump allocator.
* ForwardingHeaders/wtf/CheckedBoolean.h: Added.
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@105442 268f45cc-cd09-0410-ab3c-d52691b4dbfc
diff --git a/Source/JavaScriptCore/CMakeLists.txt b/Source/JavaScriptCore/CMakeLists.txt
index 00e0296..db56a3b 100644
--- a/Source/JavaScriptCore/CMakeLists.txt
+++ b/Source/JavaScriptCore/CMakeLists.txt
@@ -70,6 +70,7 @@
dfg/DFGThunks.cpp
heap/AllocationSpace.cpp
+ heap/BumpSpace.cpp
heap/DFGCodeBlocks.cpp
heap/Heap.cpp
heap/HandleHeap.cpp
diff --git a/Source/JavaScriptCore/ChangeLog b/Source/JavaScriptCore/ChangeLog
index 85cd0a29..a854363 100644
--- a/Source/JavaScriptCore/ChangeLog
+++ b/Source/JavaScriptCore/ChangeLog
@@ -1,3 +1,185 @@
+2012-01-19 Mark Hahnenberg <mhahnenberg@apple.com>
+
+ Implement a new allocator for backing stores
+ https://bugs.webkit.org/show_bug.cgi?id=75181
+
+ Reviewed by Filip Pizlo.
+
+ We want to move away from using fastMalloc for the backing stores for
+ some of our objects (e.g. JSArray, JSObject, JSString, etc). These backing
+ stores have a nice property in that they only have a single owner (i.e. a
+ single pointer to them at any one time). One way that we can take advantage
+ of this property is to implement a simple bump allocator/copying collector,
+ which will run alongside our normal mark/sweep collector, that only needs to
+ update the single owner pointer rather than having to redirect an arbitrary
+ number of pointers in from-space to to-space.
+
+ This plan can give us a number of benefits. We can beat fastMalloc in terms
+ of both performance and memory usage, we can track how much memory we're using
+ far more accurately than our rough estimation now through the use of
+ reportExtraMemoryCost, and we can allocate arbitrary size objects (as opposed
+ to being limited to size classes like we have been historically). This is also
+ another step toward moving away from lazy destruction, which will improve our memory footprint.
+
+ We start by creating said allocator and moving the ArrayStorage for JSArray
+ to use it rather than fastMalloc.
+
+ The design of the collector is as follows:
+ Allocation:
+ -The collector allocates 64KB chunks from the OS to use for object allocation.
+ -Each chunk contains an offset, a flag indicating if the block has been pinned,
+ and a payload, along with next and prev pointers so that they can be put in DoublyLinkedLists.
+ -Any allocation greater than 64KB gets its own separate oversize block, which
+ is managed separately from the rest.
+ -If the allocator receives a request for more than the remaining amount in the
+ current block, it grabs a fresh block.
+ -Grabbing a fresh block means grabbing one off of the global free list (which is now
+ shared between the mark/sweep allocator and the bump allocator) if there is one.
+ If there isn't a new one we do one of two things: allocate a new block from the OS
+ if we're not ready for a GC yet, or run a GC and then try again. If we still don't
+ have enough space after the GC, we allocate a new block from the OS.
+
+ Garbage collection:
+ -At the start of garbage collection during conservative stack scanning, if we encounter
+ what appears to be a pointer to a bump-allocated block of memory, we pin that block so
+ that it will not be copied for this round of collection.
+ -We also pin any oversize blocks that we encounter, which effectively doubles as a
+ "mark bit" for that block. Any oversize blocks that aren't pinned at the end of copying
+ are given back to the OS.
+ -Marking threads are now also responsible for copying bump-allocated objects to newSpace
+ -Each marking thread has a private 64KB block into which it copies bump-allocated objects that it encounters.
+ -When that block fills up, the marking thread gives it back to the allocator and requests a new one.
+ -When all marking has concluded, each thread gives back its copy block, even if it isn't full.
+ -At the conclusion of copying (which is done by the end of the marking phase), we un-pin
+ any pinned blocks and give any blocks left in from-space to the global free list.
+
+ * CMakeLists.txt:
+ * GNUmakefile.list.am:
+ * JavaScriptCore.gypi:
+ * JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj:
+ * JavaScriptCore.vcproj/WTF/WTF.vcproj:
+ * JavaScriptCore.xcodeproj/project.pbxproj:
+ * Target.pri:
+ * heap/AllocationSpace.cpp:
+ (JSC::AllocationSpace::allocateSlowCase):
+ (JSC::AllocationSpace::allocateBlock):
+ (JSC::AllocationSpace::freeBlocks):
+ * heap/AllocationSpace.h:
+ (JSC::AllocationSpace::waterMark):
+ * heap/BumpBlock.h: Added.
+ (JSC::BumpBlock::BumpBlock):
+ * heap/BumpSpace.cpp: Added.
+ (JSC::BumpSpace::tryAllocateSlowCase):
+ * heap/BumpSpace.h: Added.
+ (JSC::BumpSpace::isInCopyPhase):
+ (JSC::BumpSpace::totalMemoryAllocated):
+ (JSC::BumpSpace::totalMemoryUtilized):
+ * heap/BumpSpaceInlineMethods.h: Added.
+ (JSC::BumpSpace::BumpSpace):
+ (JSC::BumpSpace::init):
+ (JSC::BumpSpace::contains):
+ (JSC::BumpSpace::pin):
+ (JSC::BumpSpace::startedCopying):
+ (JSC::BumpSpace::doneCopying):
+ (JSC::BumpSpace::doneFillingBlock):
+ (JSC::BumpSpace::recycleBlock):
+ (JSC::BumpSpace::getFreshBlock):
+ (JSC::BumpSpace::borrowBlock):
+ (JSC::BumpSpace::addNewBlock):
+ (JSC::BumpSpace::allocateNewBlock):
+ (JSC::BumpSpace::fitsInBlock):
+ (JSC::BumpSpace::fitsInCurrentBlock):
+ (JSC::BumpSpace::tryAllocate):
+ (JSC::BumpSpace::tryAllocateOversize):
+ (JSC::BumpSpace::allocateFromBlock):
+ (JSC::BumpSpace::tryReallocate):
+ (JSC::BumpSpace::tryReallocateOversize):
+ (JSC::BumpSpace::isOversize):
+ (JSC::BumpSpace::isPinned):
+ (JSC::BumpSpace::oversizeBlockFor):
+ (JSC::BumpSpace::blockFor):
+ * heap/ConservativeRoots.cpp:
+ (JSC::ConservativeRoots::ConservativeRoots):
+ (JSC::ConservativeRoots::genericAddPointer):
+ (JSC::ConservativeRoots::add):
+ * heap/ConservativeRoots.h:
+ * heap/Heap.cpp:
+ (JSC::Heap::Heap):
+ (JSC::Heap::blockFreeingThreadMain):
+ (JSC::Heap::reportExtraMemoryCostSlowCase):
+ (JSC::Heap::getConservativeRegisterRoots):
+ (JSC::Heap::markRoots):
+ (JSC::Heap::collect):
+ (JSC::Heap::releaseFreeBlocks):
+ * heap/Heap.h:
+ (JSC::Heap::waterMark):
+ (JSC::Heap::highWaterMark):
+ (JSC::Heap::setHighWaterMark):
+ (JSC::Heap::tryAllocateStorage):
+ (JSC::Heap::tryReallocateStorage):
+ * heap/HeapBlock.h: Added.
+ (JSC::HeapBlock::HeapBlock):
+ * heap/MarkStack.cpp:
+ (JSC::MarkStackThreadSharedData::MarkStackThreadSharedData):
+ (JSC::SlotVisitor::drain):
+ (JSC::SlotVisitor::drainFromShared):
+ (JSC::SlotVisitor::startCopying):
+ (JSC::SlotVisitor::allocateNewSpace):
+ (JSC::SlotVisitor::copy):
+ (JSC::SlotVisitor::copyAndAppend):
+ (JSC::SlotVisitor::doneCopying):
+ * heap/MarkStack.h:
+ * heap/MarkedBlock.cpp:
+ (JSC::MarkedBlock::recycle):
+ (JSC::MarkedBlock::MarkedBlock):
+ * heap/MarkedBlock.h:
+ * heap/MarkedSpace.cpp:
+ (JSC::MarkedSpace::MarkedSpace):
+ * heap/MarkedSpace.h:
+ (JSC::MarkedSpace::allocate):
+ (JSC::MarkedSpace::forEachBlock):
+ (JSC::MarkedSpace::SizeClass::resetAllocator):
+ * heap/SlotVisitor.h:
+ (JSC::SlotVisitor::SlotVisitor):
+ * heap/TinyBloomFilter.h:
+ (JSC::TinyBloomFilter::reset):
+ * runtime/JSArray.cpp:
+ (JSC::JSArray::JSArray):
+ (JSC::JSArray::finishCreation):
+ (JSC::JSArray::tryFinishCreationUninitialized):
+ (JSC::JSArray::~JSArray):
+ (JSC::JSArray::enterSparseMode):
+ (JSC::JSArray::defineOwnNumericProperty):
+ (JSC::JSArray::setLengthWritable):
+ (JSC::JSArray::getOwnPropertySlotByIndex):
+ (JSC::JSArray::getOwnPropertyDescriptor):
+ (JSC::JSArray::putByIndexBeyondVectorLength):
+ (JSC::JSArray::deletePropertyByIndex):
+ (JSC::JSArray::getOwnPropertyNames):
+ (JSC::JSArray::increaseVectorLength):
+ (JSC::JSArray::unshiftCountSlowCase):
+ (JSC::JSArray::setLength):
+ (JSC::JSArray::pop):
+ (JSC::JSArray::unshiftCount):
+ (JSC::JSArray::visitChildren):
+ (JSC::JSArray::sortNumeric):
+ (JSC::JSArray::sort):
+ (JSC::JSArray::compactForSorting):
+ (JSC::JSArray::subclassData):
+ (JSC::JSArray::setSubclassData):
+ (JSC::JSArray::checkConsistency):
+ * runtime/JSArray.h:
+ (JSC::JSArray::inSparseMode):
+ (JSC::JSArray::isLengthWritable):
+ * wtf/CheckedBoolean.h: Added.
+ (CheckedBoolean::CheckedBoolean):
+ (CheckedBoolean::~CheckedBoolean):
+ (CheckedBoolean::operator bool):
+ * wtf/DoublyLinkedList.h:
+ (WTF::::push):
+ * wtf/StdLibExtras.h:
+ (WTF::isPointerAligned):
+
2012-01-19 Joi Sigurdsson <joi@chromium.org>
Enable use of precompiled headers in Chromium port on Windows.
diff --git a/Source/JavaScriptCore/GNUmakefile.list.am b/Source/JavaScriptCore/GNUmakefile.list.am
index 85593ee..27b8684 100644
--- a/Source/JavaScriptCore/GNUmakefile.list.am
+++ b/Source/JavaScriptCore/GNUmakefile.list.am
@@ -172,6 +172,10 @@
Source/JavaScriptCore/dfg/DFGVariableAccessData.h \
Source/JavaScriptCore/heap/AllocationSpace.cpp \
Source/JavaScriptCore/heap/AllocationSpace.h \
+ Source/JavaScriptCore/heap/BumpBlock.h \
+ Source/JavaScriptCore/heap/BumpSpace.cpp \
+ Source/JavaScriptCore/heap/BumpSpace.h \
+ Source/JavaScriptCore/heap/BumpSpaceInlineMethods.h \
Source/JavaScriptCore/heap/CardSet.h \
Source/JavaScriptCore/heap/ConservativeRoots.cpp \
Source/JavaScriptCore/heap/ConservativeRoots.h \
@@ -180,6 +184,7 @@
Source/JavaScriptCore/heap/Handle.h \
Source/JavaScriptCore/heap/HandleHeap.cpp \
Source/JavaScriptCore/heap/HandleHeap.h \
+ Source/JavaScriptCore/heap/HeapBlock.h \
Source/JavaScriptCore/heap/SlotVisitor.h \
Source/JavaScriptCore/heap/HandleStack.cpp \
Source/JavaScriptCore/heap/HandleStack.h \
@@ -533,6 +538,7 @@
Source/JavaScriptCore/wtf/ByteArray.cpp \
Source/JavaScriptCore/wtf/ByteArray.h \
Source/JavaScriptCore/wtf/CheckedArithmetic.h \
+ Source/JavaScriptCore/wtf/CheckedBoolean.h \
Source/JavaScriptCore/wtf/Compiler.h \
Source/JavaScriptCore/wtf/CryptographicallyRandomNumber.cpp \
Source/JavaScriptCore/wtf/CryptographicallyRandomNumber.h \
diff --git a/Source/JavaScriptCore/JavaScriptCore.gypi b/Source/JavaScriptCore/JavaScriptCore.gypi
index 1dbbf79..9a876c2 100644
--- a/Source/JavaScriptCore/JavaScriptCore.gypi
+++ b/Source/JavaScriptCore/JavaScriptCore.gypi
@@ -28,9 +28,13 @@
'assembler/MacroAssemblerCodeRef.h',
'bytecode/Opcode.h',
'heap/AllocationSpace.h',
+ 'heap/BumpBlock.h',
+ 'heap/BumpSpace.h',
+ 'heap/BumpSpaceInlineMethods.h',
'heap/ConservativeRoots.h',
'heap/Handle.h',
'heap/HandleHeap.h',
+ 'heap/HeapBlock.h',
'heap/SlotVisitor.h',
'heap/HandleStack.h',
'heap/HandleTypes.h',
@@ -144,6 +148,7 @@
'wtf/BumpPointerAllocator.h',
'wtf/ByteArray.h',
'wtf/CheckedArithmetic.h',
+ 'wtf/CheckedBoolean.h',
'wtf/Compiler.h',
'wtf/Complex.h',
'wtf/CryptographicallyRandomNumber.h',
diff --git a/Source/JavaScriptCore/JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj b/Source/JavaScriptCore/JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj
index 8fd3c8a..3862212 100644
--- a/Source/JavaScriptCore/JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj
+++ b/Source/JavaScriptCore/JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj
@@ -1974,6 +1974,22 @@
>
</File>
<File
+ RelativePath="..\..\heap\BumpBlock.h"
+ >
+ </File>
+ <File
+ RelativePath="..\..\heap\BumpSpace.cpp"
+ >
+ </File>
+ <File
+ RelativePath="..\..\heap\BumpSpace.h"
+ >
+ </File>
+ <File
+ RelativePath="..\..\heap\BumpSpaceInlineMethods.h"
+ >
+ </File>
+ <File
RelativePath="..\..\heap\ConservativeRoots.cpp"
>
</File>
@@ -2026,6 +2042,10 @@
>
</File>
<File
+ RelativePath="..\..\heap\HeapBlock.h"
+ >
+ </File>
+ <File
RelativePath="..\..\heap\Strong.h"
>
</File>
diff --git a/Source/JavaScriptCore/JavaScriptCore.vcproj/WTF/WTF.vcproj b/Source/JavaScriptCore/JavaScriptCore.vcproj/WTF/WTF.vcproj
index f72646e..c914b22 100644
--- a/Source/JavaScriptCore/JavaScriptCore.vcproj/WTF/WTF.vcproj
+++ b/Source/JavaScriptCore/JavaScriptCore.vcproj/WTF/WTF.vcproj
@@ -693,6 +693,10 @@
>
</File>
<File
+ RelativePath="..\..\wtf\CheckedBoolean.h"
+ >
+ </File>
+ <File
RelativePath="..\..\wtf\Compiler.h"
>
</File>
diff --git a/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj b/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
index e81fa16..a6f629a 100644
--- a/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
+++ b/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
@@ -710,6 +710,12 @@
C22C531313FAF6EF00B7DC0D /* strtod.cc in Sources */ = {isa = PBXBuildFile; fileRef = C22C52B913FAF6EF00B7DC0D /* strtod.cc */; };
C22C531413FAF6EF00B7DC0D /* strtod.h in Headers */ = {isa = PBXBuildFile; fileRef = C22C52BA13FAF6EF00B7DC0D /* strtod.h */; settings = {ATTRIBUTES = (Private, ); }; };
C22C531513FAF6EF00B7DC0D /* utils.h in Headers */ = {isa = PBXBuildFile; fileRef = C22C52BB13FAF6EF00B7DC0D /* utils.h */; settings = {ATTRIBUTES = (Private, ); }; };
+ C240305514B404E60079EB64 /* BumpSpace.cpp in Sources */ = {isa = PBXBuildFile; fileRef = C240305314B404C90079EB64 /* BumpSpace.cpp */; };
+ C2C8D02D14A3C6E000578E65 /* BumpSpaceInlineMethods.h in Headers */ = {isa = PBXBuildFile; fileRef = C2C8D02B14A3C6B200578E65 /* BumpSpaceInlineMethods.h */; settings = {ATTRIBUTES = (Private, ); }; };
+ C2C8D03014A3CEFC00578E65 /* BumpBlock.h in Headers */ = {isa = PBXBuildFile; fileRef = C2C8D02E14A3CEFC00578E65 /* BumpBlock.h */; settings = {ATTRIBUTES = (Private, ); }; };
+ C2C8D03114A3CEFC00578E65 /* HeapBlock.h in Headers */ = {isa = PBXBuildFile; fileRef = C2C8D02F14A3CEFC00578E65 /* HeapBlock.h */; settings = {ATTRIBUTES = (Private, ); }; };
+ C2D9CA1314BCC04600304B46 /* CheckedBoolean.h in Headers */ = {isa = PBXBuildFile; fileRef = C2D9CA1214BCC04600304B46 /* CheckedBoolean.h */; settings = {ATTRIBUTES = (Private, ); }; };
+ C2EAA3FA149A835E00FCE112 /* BumpSpace.h in Headers */ = {isa = PBXBuildFile; fileRef = C2EAA3F8149A830800FCE112 /* BumpSpace.h */; settings = {ATTRIBUTES = (Private, ); }; };
C2EE59A013FC973F009CEAFE /* DecimalNumber.h in Headers */ = {isa = PBXBuildFile; fileRef = C2EE599E13FC972A009CEAFE /* DecimalNumber.h */; settings = {ATTRIBUTES = (Private, ); }; };
C2EE59A113FC9768009CEAFE /* DecimalNumber.cpp in Sources */ = {isa = PBXBuildFile; fileRef = C2EE599D13FC972A009CEAFE /* DecimalNumber.cpp */; };
D7A46A4F1338FFEA00ED695C /* DynamicAnnotations.h in Headers */ = {isa = PBXBuildFile; fileRef = D75AF59612F8CB9500FC0ADF /* DynamicAnnotations.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -1475,6 +1481,12 @@
C22C52B913FAF6EF00B7DC0D /* strtod.cc */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = strtod.cc; sourceTree = "<group>"; };
C22C52BA13FAF6EF00B7DC0D /* strtod.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = strtod.h; sourceTree = "<group>"; };
C22C52BB13FAF6EF00B7DC0D /* utils.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = utils.h; sourceTree = "<group>"; };
+ C240305314B404C90079EB64 /* BumpSpace.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = BumpSpace.cpp; sourceTree = "<group>"; };
+ C2C8D02B14A3C6B200578E65 /* BumpSpaceInlineMethods.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BumpSpaceInlineMethods.h; sourceTree = "<group>"; };
+ C2C8D02E14A3CEFC00578E65 /* BumpBlock.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BumpBlock.h; sourceTree = "<group>"; };
+ C2C8D02F14A3CEFC00578E65 /* HeapBlock.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapBlock.h; sourceTree = "<group>"; };
+ C2D9CA1214BCC04600304B46 /* CheckedBoolean.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CheckedBoolean.h; sourceTree = "<group>"; };
+ C2EAA3F8149A830800FCE112 /* BumpSpace.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BumpSpace.h; sourceTree = "<group>"; };
C2EE599D13FC972A009CEAFE /* DecimalNumber.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = DecimalNumber.cpp; sourceTree = "<group>"; };
C2EE599E13FC972A009CEAFE /* DecimalNumber.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DecimalNumber.h; sourceTree = "<group>"; };
D21202280AD4310C00ED79B6 /* DateConversion.cpp */ = {isa = PBXFileReference; fileEncoding = 30; lastKnownFileType = sourcecode.cpp.cpp; path = DateConversion.cpp; sourceTree = "<group>"; };
@@ -1745,6 +1757,10 @@
children = (
A70456AE1427FB030037DA68 /* AllocationSpace.cpp */,
A70456AF1427FB150037DA68 /* AllocationSpace.h */,
+ C2C8D02E14A3CEFC00578E65 /* BumpBlock.h */,
+ C240305314B404C90079EB64 /* BumpSpace.cpp */,
+ C2EAA3F8149A830800FCE112 /* BumpSpace.h */,
+ C2C8D02B14A3C6B200578E65 /* BumpSpaceInlineMethods.h */,
A7521E121429169A003C8D0C /* CardSet.h */,
146B14DB12EB5B12001BEC1B /* ConservativeRoots.cpp */,
149DAAF212EB559D0083B12B /* ConservativeRoots.h */,
@@ -1758,6 +1774,7 @@
146FA5A81378F6B0003627A3 /* HandleTypes.h */,
14BA7A9513AADFF8005B7C2C /* Heap.cpp */,
14BA7A9613AADFF8005B7C2C /* Heap.h */,
+ C2C8D02F14A3CEFC00578E65 /* HeapBlock.h */,
14F97446138C853E00DA1C67 /* HeapRootVisitor.h */,
0F431736146BAC65007E3890 /* ListableHandler.h */,
142E3130134FF0A600AFADB5 /* Local.h */,
@@ -2083,6 +2100,7 @@
96DD73780F9DA3100027FBCC /* VMTags.h */,
86D08D5111793613006E5ED0 /* WTFThreadData.cpp */,
86D08D5211793613006E5ED0 /* WTFThreadData.h */,
+ C2D9CA1214BCC04600304B46 /* CheckedBoolean.h */,
);
path = wtf;
sourceTree = "<group>";
@@ -2613,6 +2631,7 @@
86D3B2C410156BDE002865E7 /* ARMAssembler.h in Headers */,
86ADD1450FDDEA980006EEC2 /* ARMv7Assembler.h in Headers */,
A73BE169148420520091204B /* ArrayBuffer.h in Headers */,
+ C2D9CA1314BCC04600304B46 /* CheckedBoolean.h in Headers */,
A73BE16B148420520091204B /* ArrayBufferView.h in Headers */,
BC18C3E60E16F5CD00B34460 /* ArrayConstructor.h in Headers */,
BC18C3E70E16F5CD00B34460 /* ArrayPrototype.h in Headers */,
@@ -2639,7 +2658,10 @@
E4D8CEFB12FC439600BC9F5A /* BloomFilter.h in Headers */,
BC18C3EC0E16F5CD00B34460 /* BooleanObject.h in Headers */,
0FD82E85141F3FE300179C94 /* BoundsCheckedPointer.h in Headers */,
+ C2C8D03014A3CEFC00578E65 /* BumpBlock.h in Headers */,
86676D5211FED9BC004B6863 /* BumpPointerAllocator.h in Headers */,
+ C2EAA3FA149A835E00FCE112 /* BumpSpace.h in Headers */,
+ C2C8D02D14A3C6E000578E65 /* BumpSpaceInlineMethods.h in Headers */,
A7A1F7AD0F252B3C00E184E2 /* ByteArray.h in Headers */,
969A07230ED1CE3300F1F681 /* BytecodeGenerator.h in Headers */,
C22C52F613FAF6EF00B7DC0D /* cached-powers.h in Headers */,
@@ -2751,6 +2773,7 @@
BC18C40D0E16F5CD00B34460 /* HashTable.h in Headers */,
BC18C40E0E16F5CD00B34460 /* HashTraits.h in Headers */,
14BA7A9813AADFF8005B7C2C /* Heap.h in Headers */,
+ C2C8D03114A3CEFC00578E65 /* HeapBlock.h in Headers */,
14F97447138C853E00DA1C67 /* HeapRootVisitor.h in Headers */,
7186A6EC13100BA5004479E1 /* HexNumber.h in Headers */,
BC18C40F0E16F5CD00B34460 /* Identifier.h in Headers */,
@@ -3373,6 +3396,7 @@
14280863107EC11A0013E7B2 /* BooleanConstructor.cpp in Sources */,
14280864107EC11A0013E7B2 /* BooleanObject.cpp in Sources */,
14280865107EC11A0013E7B2 /* BooleanPrototype.cpp in Sources */,
+ C240305514B404E60079EB64 /* BumpSpace.cpp in Sources */,
A7A1F7AC0F252B3C00E184E2 /* ByteArray.cpp in Sources */,
148F21AA107EC53A0042EC2C /* BytecodeGenerator.cpp in Sources */,
C22C52F513FAF6EF00B7DC0D /* cached-powers.cc in Sources */,
diff --git a/Source/JavaScriptCore/Target.pri b/Source/JavaScriptCore/Target.pri
index 6ea82d1..002b48d 100644
--- a/Source/JavaScriptCore/Target.pri
+++ b/Source/JavaScriptCore/Target.pri
@@ -62,6 +62,7 @@
bytecompiler/BytecodeGenerator.cpp \
bytecompiler/NodesCodegen.cpp \
heap/AllocationSpace.cpp \
+ heap/BumpSpace.cpp \
heap/ConservativeRoots.cpp \
heap/DFGCodeBlocks.cpp \
heap/HandleHeap.cpp \
diff --git a/Source/JavaScriptCore/heap/AllocationSpace.cpp b/Source/JavaScriptCore/heap/AllocationSpace.cpp
index e363de2..290f2a2 100644
--- a/Source/JavaScriptCore/heap/AllocationSpace.cpp
+++ b/Source/JavaScriptCore/heap/AllocationSpace.cpp
@@ -29,8 +29,6 @@
#include "Heap.h"
-#define COLLECT_ON_EVERY_ALLOCATION 0
-
namespace JSC {
inline void* AllocationSpace::tryAllocate(MarkedSpace::SizeClass& sizeClass)
@@ -59,7 +57,7 @@
#if ENABLE(GGC)
m_markedSpace.nurseryWaterMark() < m_heap->m_minBytesPerCycle
#else
- m_markedSpace.waterMark() < m_markedSpace.highWaterMark()
+ m_heap->waterMark() < m_heap->highWaterMark()
#endif
) || !m_heap->m_isSafeToCollect)
allocationEffort = AllocationMustSucceed;
@@ -81,7 +79,7 @@
if (result)
return result;
- ASSERT(m_markedSpace.waterMark() < m_markedSpace.highWaterMark());
+ ASSERT(m_heap->waterMark() < m_heap->highWaterMark());
m_markedSpace.addBlock(sizeClass, allocateBlock(sizeClass.cellSize, AllocationMustSucceed));
@@ -90,21 +88,21 @@
return result;
}
-MarkedBlock* AllocationSpace::allocateBlock(size_t cellSize, AllocationSpace::AllocationEffort allocationEffort)
+MarkedBlock* AllocationSpace::allocateBlock(size_t cellSize, AllocationEffort allocationEffort)
{
MarkedBlock* block;
{
MutexLocker locker(m_heap->m_freeBlockLock);
if (m_heap->m_numberOfFreeBlocks) {
- block = m_heap->m_freeBlocks.removeHead();
+ block = static_cast<MarkedBlock*>(m_heap->m_freeBlocks.removeHead());
ASSERT(block);
m_heap->m_numberOfFreeBlocks--;
} else
block = 0;
}
if (block)
- block = MarkedBlock::recycle(block, cellSize);
+ block = MarkedBlock::recycle(block, m_heap, cellSize);
else if (allocationEffort == AllocationCanFail)
return 0;
else
@@ -119,7 +117,7 @@
{
MarkedBlock* next;
for (MarkedBlock* block = head; block; block = next) {
- next = block->next();
+ next = static_cast<MarkedBlock*>(block->next());
m_blocks.remove(block);
block->sweep();
diff --git a/Source/JavaScriptCore/heap/AllocationSpace.h b/Source/JavaScriptCore/heap/AllocationSpace.h
index 01c339b..45a6b5a 100644
--- a/Source/JavaScriptCore/heap/AllocationSpace.h
+++ b/Source/JavaScriptCore/heap/AllocationSpace.h
@@ -48,8 +48,7 @@
MarkedBlockSet& blocks() { return m_blocks; }
MarkedSpace::SizeClass& sizeClassFor(size_t bytes) { return m_markedSpace.sizeClassFor(bytes); }
- void setHighWaterMark(size_t bytes) { m_markedSpace.setHighWaterMark(bytes); }
- size_t highWaterMark() { return m_markedSpace.highWaterMark(); }
+ size_t waterMark() { return m_markedSpace.waterMark(); }
#if ENABLE(GGC)
void gatherDirtyCells(MarkedBlock::DirtyCellVector&);
@@ -68,8 +67,8 @@
void shrink();
private:
- enum AllocationEffort { AllocationMustSucceed, AllocationCanFail };
-
+ enum AllocationEffort { AllocationCanFail, AllocationMustSucceed };
+
void* allocate(MarkedSpace::SizeClass&);
void* tryAllocate(MarkedSpace::SizeClass&);
JS_EXPORT_PRIVATE void* allocateSlowCase(MarkedSpace::SizeClass&);
diff --git a/Source/JavaScriptCore/heap/BumpBlock.h b/Source/JavaScriptCore/heap/BumpBlock.h
new file mode 100644
index 0000000..b9f271c
--- /dev/null
+++ b/Source/JavaScriptCore/heap/BumpBlock.h
@@ -0,0 +1,53 @@
+/*
+ * Copyright (C) 2011 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef BumpBlock_h
+#define BumpBlock_h
+
+#include "HeapBlock.h"
+
+namespace JSC {
+
+class BumpSpace;
+
+class BumpBlock : public HeapBlock {
+ friend class BumpSpace;
+public:
+ BumpBlock(PageAllocationAligned& allocation)
+ : HeapBlock(allocation)
+ , m_offset(m_payload)
+ , m_isPinned(false)
+ {
+ }
+
+private:
+ void* m_offset;
+ uintptr_t m_isPinned;
+ char m_payload[1];
+};
+
+} // namespace JSC
+
+#endif
diff --git a/Source/JavaScriptCore/heap/BumpSpace.cpp b/Source/JavaScriptCore/heap/BumpSpace.cpp
new file mode 100644
index 0000000..4eb0284
--- /dev/null
+++ b/Source/JavaScriptCore/heap/BumpSpace.cpp
@@ -0,0 +1,49 @@
+/*
+ * Copyright (C) 2011 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "BumpSpace.h"
+
+#include "BumpSpaceInlineMethods.h"
+
+namespace JSC {
+
+CheckedBoolean BumpSpace::tryAllocateSlowCase(size_t bytes, void** outPtr)
+{
+ if (isOversize(bytes))
+ return tryAllocateOversize(bytes, outPtr);
+
+ m_totalMemoryUtilized += static_cast<size_t>(static_cast<char*>(m_currentBlock->m_offset) - m_currentBlock->m_payload);
+ if (!addNewBlock()) {
+ *outPtr = 0;
+ return false;
+ }
+ m_toSpaceFilter.add(reinterpret_cast<Bits>(m_currentBlock));
+ m_toSpaceSet.add(m_currentBlock);
+ *outPtr = allocateFromBlock(m_currentBlock, bytes);
+ return true;
+}
+
+} // namespace JSC
diff --git a/Source/JavaScriptCore/heap/BumpSpace.h b/Source/JavaScriptCore/heap/BumpSpace.h
new file mode 100644
index 0000000..1ffcb46
--- /dev/null
+++ b/Source/JavaScriptCore/heap/BumpSpace.h
@@ -0,0 +1,126 @@
+/*
+ * Copyright (C) 2011 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef BumpSpace_h
+#define BumpSpace_h
+
+#include "TinyBloomFilter.h"
+#include <wtf/Assertions.h>
+#include <wtf/CheckedBoolean.h>
+#include <wtf/DoublyLinkedList.h>
+#include <wtf/HashSet.h>
+#include <wtf/OSAllocator.h>
+#include <wtf/PageAllocationAligned.h>
+#include <wtf/StdLibExtras.h>
+#include <wtf/ThreadingPrimitives.h>
+
+namespace JSC {
+
+class Heap;
+class BumpBlock;
+class HeapBlock;
+
+class BumpSpace {
+ friend class SlotVisitor;
+public:
+ BumpSpace(Heap*);
+ void init();
+
+ CheckedBoolean tryAllocate(size_t, void**);
+ CheckedBoolean tryReallocate(void**, size_t, size_t);
+
+ void startedCopying();
+ void doneCopying();
+ bool isInCopyPhase() { return m_inCopyingPhase; }
+
+ void pin(BumpBlock*);
+ bool isPinned(void*);
+
+ bool contains(void*, BumpBlock*&);
+
+ size_t totalMemoryAllocated() { return m_totalMemoryAllocated; }
+ size_t totalMemoryUtilized() { return m_totalMemoryUtilized; }
+
+ static BumpBlock* blockFor(void*);
+
+private:
+ enum AllocationEffort { AllocationCanFail, AllocationMustSucceed };
+
+ CheckedBoolean tryAllocateSlowCase(size_t, void**);
+ CheckedBoolean addNewBlock();
+ CheckedBoolean allocateNewBlock(BumpBlock**);
+ bool fitsInCurrentBlock(size_t);
+
+ static void* allocateFromBlock(BumpBlock*, size_t);
+ CheckedBoolean tryAllocateOversize(size_t, void**);
+ CheckedBoolean tryReallocateOversize(void**, size_t, size_t);
+
+ static bool isOversize(size_t);
+
+ CheckedBoolean borrowBlock(BumpBlock**);
+ CheckedBoolean getFreshBlock(AllocationEffort, BumpBlock**);
+ void doneFillingBlock(BumpBlock*);
+ void recycleBlock(BumpBlock*);
+ static bool fitsInBlock(BumpBlock*, size_t);
+ static BumpBlock* oversizeBlockFor(void* ptr);
+
+ Heap* m_heap;
+
+ BumpBlock* m_currentBlock;
+
+ TinyBloomFilter m_toSpaceFilter;
+ TinyBloomFilter m_oversizeFilter;
+ HashSet<BumpBlock*> m_toSpaceSet;
+
+ Mutex m_toSpaceLock;
+ Mutex m_memoryStatsLock;
+
+ DoublyLinkedList<HeapBlock>* m_toSpace;
+ DoublyLinkedList<HeapBlock>* m_fromSpace;
+
+ DoublyLinkedList<HeapBlock> m_blocks1;
+ DoublyLinkedList<HeapBlock> m_blocks2;
+ DoublyLinkedList<HeapBlock> m_oversizeBlocks;
+
+ size_t m_totalMemoryAllocated;
+ size_t m_totalMemoryUtilized;
+
+ bool m_inCopyingPhase;
+
+ Mutex m_loanedBlocksLock;
+ ThreadCondition m_loanedBlocksCondition;
+ size_t m_numberOfLoanedBlocks;
+
+ static const size_t s_blockSize = 64 * KB;
+ static const size_t s_maxAllocationSize = 32 * KB;
+ static const size_t s_pageSize = 4 * KB;
+ static const size_t s_pageMask = ~(s_pageSize - 1);
+ static const size_t s_initialBlockNum = 16;
+ static const size_t s_blockMask = ~(s_blockSize - 1);
+};
+
+} // namespace JSC
+
+#endif
diff --git a/Source/JavaScriptCore/heap/BumpSpaceInlineMethods.h b/Source/JavaScriptCore/heap/BumpSpaceInlineMethods.h
new file mode 100644
index 0000000..3454631
--- /dev/null
+++ b/Source/JavaScriptCore/heap/BumpSpaceInlineMethods.h
@@ -0,0 +1,400 @@
+/*
+ * Copyright (C) 2011 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef BumpSpaceInlineMethods_h
+#define BumpSpaceInlineMethods_h
+
+#include "BumpBlock.h"
+#include "BumpSpace.h"
+#include "Heap.h"
+#include "HeapBlock.h"
+#include "JSGlobalData.h"
+#include <wtf/CheckedBoolean.h>
+
+namespace JSC {
+
+inline BumpSpace::BumpSpace(Heap* heap)
+ : m_heap(heap)
+ , m_currentBlock(0)
+ , m_toSpace(0)
+ , m_fromSpace(0)
+ , m_totalMemoryAllocated(0)
+ , m_totalMemoryUtilized(0)
+ , m_inCopyingPhase(false)
+ , m_numberOfLoanedBlocks(0)
+{
+}
+
+inline void BumpSpace::init()
+{
+ m_toSpace = &m_blocks1;
+ m_fromSpace = &m_blocks2;
+
+ m_totalMemoryAllocated += s_blockSize * s_initialBlockNum;
+
+ if (!addNewBlock())
+ CRASH();
+}
+
+inline bool BumpSpace::contains(void* ptr, BumpBlock*& result)
+{
+ BumpBlock* block = blockFor(ptr);
+ result = block;
+ return !m_toSpaceFilter.ruleOut(reinterpret_cast<Bits>(block)) && m_toSpaceSet.contains(block);
+}
+
+inline void BumpSpace::pin(BumpBlock* block)
+{
+ block->m_isPinned = true;
+}
+
+inline void BumpSpace::startedCopying()
+{
+ DoublyLinkedList<HeapBlock>* temp = m_fromSpace;
+ m_fromSpace = m_toSpace;
+ m_toSpace = temp;
+
+ m_toSpaceFilter.reset();
+
+ m_totalMemoryUtilized = 0;
+
+ ASSERT(!m_inCopyingPhase);
+ ASSERT(!m_numberOfLoanedBlocks);
+ m_inCopyingPhase = true;
+}
+
+inline void BumpSpace::doneCopying()
+{
+ {
+ MutexLocker locker(m_loanedBlocksLock);
+ while (m_numberOfLoanedBlocks > 0)
+ m_loanedBlocksCondition.wait(m_loanedBlocksLock);
+ }
+
+ ASSERT(m_inCopyingPhase);
+ m_inCopyingPhase = false;
+ while (!m_fromSpace->isEmpty()) {
+ BumpBlock* block = static_cast<BumpBlock*>(m_fromSpace->removeHead());
+ if (block->m_isPinned) {
+ block->m_isPinned = false;
+ m_toSpace->push(block);
+ continue;
+ }
+
+ m_toSpaceSet.remove(block);
+ {
+ MutexLocker locker(m_heap->m_freeBlockLock);
+ m_heap->m_freeBlocks.push(block);
+ m_heap->m_numberOfFreeBlocks++;
+ }
+ }
+
+ BumpBlock* curr = static_cast<BumpBlock*>(m_oversizeBlocks.head());
+ while (curr) {
+ BumpBlock* next = static_cast<BumpBlock*>(curr->next());
+ if (!curr->m_isPinned) {
+ m_oversizeBlocks.remove(curr);
+ m_totalMemoryAllocated -= curr->m_allocation.size();
+ m_totalMemoryUtilized -= curr->m_allocation.size() - sizeof(BumpBlock);
+ curr->m_allocation.deallocate();
+ } else
+ curr->m_isPinned = false;
+ curr = next;
+ }
+
+ if (!(m_currentBlock = static_cast<BumpBlock*>(m_toSpace->head())))
+ if (!addNewBlock())
+ CRASH();
+}
+
+inline void BumpSpace::doneFillingBlock(BumpBlock* block)
+{
+ ASSERT(block);
+ ASSERT(block->m_offset < reinterpret_cast<char*>(block) + s_blockSize);
+ ASSERT(m_inCopyingPhase);
+
+ if (block->m_offset == block->m_payload) {
+ recycleBlock(block);
+ return;
+ }
+
+ {
+ MutexLocker locker(m_toSpaceLock);
+ m_toSpace->push(block);
+ m_toSpaceSet.add(block);
+ m_toSpaceFilter.add(reinterpret_cast<Bits>(block));
+ }
+
+ {
+ MutexLocker locker(m_memoryStatsLock);
+ m_totalMemoryUtilized += static_cast<size_t>(static_cast<char*>(block->m_offset) - block->m_payload);
+ }
+
+ {
+ MutexLocker locker(m_loanedBlocksLock);
+ ASSERT(m_numberOfLoanedBlocks > 0);
+ m_numberOfLoanedBlocks--;
+ if (!m_numberOfLoanedBlocks)
+ m_loanedBlocksCondition.signal();
+ }
+}
+
+inline void BumpSpace::recycleBlock(BumpBlock* block)
+{
+ {
+ MutexLocker locker(m_heap->m_freeBlockLock);
+ m_heap->m_freeBlocks.push(block);
+ m_heap->m_numberOfFreeBlocks++;
+ }
+
+ {
+ MutexLocker locker(m_loanedBlocksLock);
+ ASSERT(m_numberOfLoanedBlocks > 0);
+ m_numberOfLoanedBlocks--;
+ if (!m_numberOfLoanedBlocks)
+ m_loanedBlocksCondition.signal();
+ }
+}
+
+inline CheckedBoolean BumpSpace::getFreshBlock(AllocationEffort allocationEffort, BumpBlock** outBlock)
+{
+ HeapBlock* heapBlock = 0;
+ BumpBlock* block = 0;
+ {
+ MutexLocker locker(m_heap->m_freeBlockLock);
+ if (!m_heap->m_freeBlocks.isEmpty()) {
+ heapBlock = m_heap->m_freeBlocks.removeHead();
+ m_heap->m_numberOfFreeBlocks--;
+ }
+ }
+ if (heapBlock)
+ block = new (NotNull, heapBlock) BumpBlock(heapBlock->m_allocation);
+ else if (allocationEffort == AllocationMustSucceed) {
+ if (!allocateNewBlock(&block)) {
+ *outBlock = 0;
+ ASSERT_NOT_REACHED();
+ return false;
+ }
+ } else {
+ ASSERT(allocationEffort == AllocationCanFail);
+ if (m_heap->waterMark() >= m_heap->highWaterMark() && m_heap->m_isSafeToCollect)
+ m_heap->collect(Heap::DoNotSweep);
+
+ if (!getFreshBlock(AllocationMustSucceed, &block)) {
+ *outBlock = 0;
+ ASSERT_NOT_REACHED();
+ return false;
+ }
+ }
+ ASSERT(block);
+ ASSERT(isPointerAligned(block->m_offset));
+ *outBlock = block;
+ return true;
+}
+
+inline CheckedBoolean BumpSpace::borrowBlock(BumpBlock** outBlock)
+{
+ BumpBlock* block = 0;
+ if (!getFreshBlock(AllocationMustSucceed, &block)) {
+ *outBlock = 0;
+ return false;
+ }
+
+ ASSERT(m_inCopyingPhase);
+ MutexLocker locker(m_loanedBlocksLock);
+ m_numberOfLoanedBlocks++;
+
+ ASSERT(block->m_offset == block->m_payload);
+ *outBlock = block;
+ return true;
+}
+
+inline CheckedBoolean BumpSpace::addNewBlock()
+{
+ BumpBlock* block = 0;
+ if (!getFreshBlock(AllocationCanFail, &block))
+ return false;
+
+ m_toSpace->push(block);
+ m_currentBlock = block;
+ return true;
+}
+
+inline CheckedBoolean BumpSpace::allocateNewBlock(BumpBlock** outBlock)
+{
+ PageAllocationAligned allocation = PageAllocationAligned::allocate(s_blockSize, s_blockSize, OSAllocator::JSGCHeapPages);
+ if (!static_cast<bool>(allocation)) {
+ *outBlock = 0;
+ return false;
+ }
+
+ {
+ MutexLocker locker(m_memoryStatsLock);
+ m_totalMemoryAllocated += s_blockSize;
+ }
+
+ *outBlock = new (NotNull, allocation.base()) BumpBlock(allocation);
+ return true;
+}
+
+inline bool BumpSpace::fitsInBlock(BumpBlock* block, size_t bytes)
+{
+ return static_cast<char*>(block->m_offset) + bytes < reinterpret_cast<char*>(block) + s_blockSize && static_cast<char*>(block->m_offset) + bytes > block->m_offset;
+}
+
+inline bool BumpSpace::fitsInCurrentBlock(size_t bytes)
+{
+ return fitsInBlock(m_currentBlock, bytes);
+}
+
+inline CheckedBoolean BumpSpace::tryAllocate(size_t bytes, void** outPtr)
+{
+ ASSERT(!m_heap->globalData()->isInitializingObject());
+
+ if (isOversize(bytes) || !fitsInCurrentBlock(bytes))
+ return tryAllocateSlowCase(bytes, outPtr);
+
+ *outPtr = allocateFromBlock(m_currentBlock, bytes);
+ return true;
+}
+
+inline CheckedBoolean BumpSpace::tryAllocateOversize(size_t bytes, void** outPtr)
+{
+ ASSERT(isOversize(bytes));
+
+ size_t blockSize = WTF::roundUpToMultipleOf<s_pageSize>(sizeof(BumpBlock) + bytes);
+ PageAllocationAligned allocation = PageAllocationAligned::allocate(blockSize, s_pageSize, OSAllocator::JSGCHeapPages);
+ if (!static_cast<bool>(allocation)) {
+ *outPtr = 0;
+ return false;
+ }
+ BumpBlock* block = new (NotNull, allocation.base()) BumpBlock(allocation);
+ m_oversizeBlocks.push(block);
+ ASSERT(isPointerAligned(block->m_offset));
+
+ m_oversizeFilter.add(reinterpret_cast<Bits>(block));
+
+ m_totalMemoryAllocated += blockSize;
+ m_totalMemoryUtilized += bytes;
+
+ *outPtr = block->m_offset;
+ return true;
+}
+
+inline void* BumpSpace::allocateFromBlock(BumpBlock* block, size_t bytes)
+{
+ ASSERT(!isOversize(bytes));
+ ASSERT(fitsInBlock(block, bytes));
+ ASSERT(isPointerAligned(block->m_offset));
+
+ void* ptr = block->m_offset;
+ ASSERT(block->m_offset >= block->m_payload && block->m_offset < reinterpret_cast<char*>(block) + s_blockSize);
+ block->m_offset = static_cast<void*>((static_cast<char*>(ptr) + bytes));
+ ASSERT(block->m_offset >= block->m_payload && block->m_offset < reinterpret_cast<char*>(block) + s_blockSize);
+
+ ASSERT(isPointerAligned(ptr));
+ return ptr;
+}
+
+inline CheckedBoolean BumpSpace::tryReallocate(void** ptr, size_t oldSize, size_t newSize)
+{
+ if (oldSize >= newSize)
+ return true;
+
+ void* oldPtr = *ptr;
+ ASSERT(!m_heap->globalData()->isInitializingObject());
+
+ if (isOversize(oldSize) || isOversize(newSize))
+ return tryReallocateOversize(ptr, oldSize, newSize);
+
+ if (static_cast<char*>(oldPtr) + oldSize == m_currentBlock->m_offset && oldPtr > m_currentBlock && oldPtr < reinterpret_cast<char*>(m_currentBlock) + s_blockSize) {
+ m_currentBlock->m_offset = oldPtr;
+ if (fitsInCurrentBlock(newSize)) {
+ m_totalMemoryUtilized += newSize - oldSize;
+ return allocateFromBlock(m_currentBlock, newSize);
+ }
+ }
+ m_totalMemoryUtilized -= oldSize;
+
+ void* result = 0;
+ if (!tryAllocate(newSize, &result)) {
+ *ptr = 0;
+ return false;
+ }
+ memcpy(result, oldPtr, oldSize);
+ *ptr = result;
+ return true;
+}
+
+inline CheckedBoolean BumpSpace::tryReallocateOversize(void** ptr, size_t oldSize, size_t newSize)
+{
+ ASSERT(isOversize(oldSize) || isOversize(newSize));
+ ASSERT(newSize > oldSize);
+
+ void* oldPtr = *ptr;
+
+ void* newPtr = 0;
+ if (!tryAllocateOversize(newSize, &newPtr)) {
+ *ptr = 0;
+ return false;
+ }
+ memcpy(newPtr, oldPtr, oldSize);
+
+ if (isOversize(oldSize)) {
+ BumpBlock* oldBlock = oversizeBlockFor(oldPtr);
+ m_oversizeBlocks.remove(oldBlock);
+ oldBlock->m_allocation.deallocate();
+ m_totalMemoryAllocated -= oldSize + sizeof(BumpBlock);
+ }
+
+ m_totalMemoryUtilized -= oldSize;
+
+ *ptr = newPtr;
+ return true;
+}
+
+inline bool BumpSpace::isOversize(size_t bytes)
+{
+ return bytes > s_maxAllocationSize;
+}
+
+inline bool BumpSpace::isPinned(void* ptr)
+{
+ return blockFor(ptr)->m_isPinned;
+}
+
+inline BumpBlock* BumpSpace::oversizeBlockFor(void* ptr)
+{
+ return reinterpret_cast<BumpBlock*>(reinterpret_cast<size_t>(ptr) & s_pageMask);
+}
+
+inline BumpBlock* BumpSpace::blockFor(void* ptr)
+{
+ return reinterpret_cast<BumpBlock*>(reinterpret_cast<size_t>(ptr) & s_blockMask);
+}
+
+} // namespace JSC
+
+#endif
diff --git a/Source/JavaScriptCore/heap/ConservativeRoots.cpp b/Source/JavaScriptCore/heap/ConservativeRoots.cpp
index 05c668c..a509f06 100644
--- a/Source/JavaScriptCore/heap/ConservativeRoots.cpp
+++ b/Source/JavaScriptCore/heap/ConservativeRoots.cpp
@@ -26,6 +26,8 @@
#include "config.h"
#include "ConservativeRoots.h"
+#include "BumpSpace.h"
+#include "BumpSpaceInlineMethods.h"
#include "CodeBlock.h"
#include "DFGCodeBlocks.h"
#include "JSCell.h"
@@ -34,16 +36,12 @@
namespace JSC {
-inline bool isPointerAligned(void* p)
-{
- return !((intptr_t)(p) & (sizeof(char*) - 1));
-}
-
-ConservativeRoots::ConservativeRoots(const MarkedBlockSet* blocks)
+ConservativeRoots::ConservativeRoots(const MarkedBlockSet* blocks, BumpSpace* bumpSpace)
: m_roots(m_inlineRoots)
, m_size(0)
, m_capacity(inlineCapacity)
, m_blocks(blocks)
+ , m_bumpSpace(bumpSpace)
{
}
@@ -74,6 +72,10 @@
{
markHook.mark(p);
+ BumpBlock* block;
+ if (m_bumpSpace->contains(p, block))
+ m_bumpSpace->pin(block);
+
MarkedBlock* candidate = MarkedBlock::blockFor(p);
if (filter.ruleOut(reinterpret_cast<Bits>(candidate))) {
ASSERT(!candidate || !m_blocks->set().contains(candidate));
@@ -110,8 +112,8 @@
void ConservativeRoots::add(void* begin, void* end)
{
- DummyMarkHook dummyMarkHook;
- genericAddSpan(begin, end, dummyMarkHook);
+ DummyMarkHook hook;
+ genericAddSpan(begin, end, hook);
}
void ConservativeRoots::add(void* begin, void* end, DFGCodeBlocks& dfgCodeBlocks)
diff --git a/Source/JavaScriptCore/heap/ConservativeRoots.h b/Source/JavaScriptCore/heap/ConservativeRoots.h
index 86dfc58..40b0996 100644
--- a/Source/JavaScriptCore/heap/ConservativeRoots.h
+++ b/Source/JavaScriptCore/heap/ConservativeRoots.h
@@ -38,7 +38,7 @@
class ConservativeRoots {
public:
- ConservativeRoots(const MarkedBlockSet*);
+ ConservativeRoots(const MarkedBlockSet*, BumpSpace*);
~ConservativeRoots();
void add(void* begin, void* end);
@@ -63,6 +63,7 @@
size_t m_size;
size_t m_capacity;
const MarkedBlockSet* m_blocks;
+ BumpSpace* m_bumpSpace;
JSCell* m_inlineRoots[inlineCapacity];
};
diff --git a/Source/JavaScriptCore/heap/Heap.cpp b/Source/JavaScriptCore/heap/Heap.cpp
index 61eba08..a213608 100644
--- a/Source/JavaScriptCore/heap/Heap.cpp
+++ b/Source/JavaScriptCore/heap/Heap.cpp
@@ -21,6 +21,8 @@
#include "config.h"
#include "Heap.h"
+#include "BumpSpace.h"
+#include "BumpSpaceInlineMethods.h"
#include "CodeBlock.h"
#include "ConservativeRoots.h"
#include "GCActivityCallback.h"
@@ -311,8 +313,11 @@
: m_heapSize(heapSize)
, m_minBytesPerCycle(heapSizeForHint(heapSize))
, m_lastFullGCSize(0)
+ , m_waterMark(0)
+ , m_highWaterMark(m_minBytesPerCycle)
, m_operationInProgress(NoOperation)
, m_objectSpace(this)
+ , m_storageSpace(this)
, m_blockFreeingThreadShouldQuit(false)
, m_extraCost(0)
, m_markListSet(0)
@@ -324,12 +329,12 @@
, m_isSafeToCollect(false)
, m_globalData(globalData)
{
- m_objectSpace.setHighWaterMark(m_minBytesPerCycle);
(*m_activityCallback)();
m_numberOfFreeBlocks = 0;
m_blockFreeingThread = createThread(blockFreeingThreadStartFunc, this, "JavaScriptCore::BlockFree");
ASSERT(m_blockFreeingThread);
+ m_storageSpace.init();
}
Heap::~Heap()
@@ -433,7 +438,7 @@
if (m_numberOfFreeBlocks <= desiredNumberOfFreeBlocks)
block = 0;
else {
- block = m_freeBlocks.removeHead();
+ block = static_cast<MarkedBlock*>(m_freeBlocks.removeHead());
ASSERT(block);
m_numberOfFreeBlocks--;
}
@@ -460,7 +465,7 @@
// if a large value survives one garbage collection, there is not much point to
// collecting more frequently as long as it stays alive.
- if (m_extraCost > maxExtraCost && m_extraCost > m_objectSpace.highWaterMark() / 2)
+ if (m_extraCost > maxExtraCost && m_extraCost > highWaterMark() / 2)
collectAllGarbage();
m_extraCost += cost;
}
@@ -509,7 +514,7 @@
ASSERT_UNUSED(tempVector, tempVector == m_tempSortingVectors.last());
m_tempSortingVectors.removeLast();
}
-
+
void Heap::markTempSortVectors(HeapRootVisitor& heapRootVisitor)
{
typedef Vector<Vector<ValueStringPair>* > VectorOfValueStringVectors;
@@ -547,7 +552,7 @@
if (m_operationInProgress != NoOperation)
CRASH();
m_operationInProgress = Collection;
- ConservativeRoots registerFileRoots(&m_objectSpace.blocks());
+ ConservativeRoots registerFileRoots(&m_objectSpace.blocks(), &m_storageSpace);
registerFile().gatherConservativeRoots(registerFileRoots);
size_t registerFileRootCount = registerFileRoots.size();
JSCell** registerRoots = registerFileRoots.roots();
@@ -573,13 +578,13 @@
// We gather conservative roots before clearing mark bits because conservative
// gathering uses the mark bits to determine whether a reference is valid.
- ConservativeRoots machineThreadRoots(&m_objectSpace.blocks());
+ ConservativeRoots machineThreadRoots(&m_objectSpace.blocks(), &m_storageSpace);
{
GCPHASE(GatherConservativeRoots);
m_machineThreads.gatherConservativeRoots(machineThreadRoots, &dummy);
}
- ConservativeRoots registerFileRoots(&m_objectSpace.blocks());
+ ConservativeRoots registerFileRoots(&m_objectSpace.blocks(), &m_storageSpace);
m_dfgCodeBlocks.clearMarks();
{
GCPHASE(GatherRegisterFileRoots);
@@ -597,6 +602,7 @@
clearMarks();
}
+ m_storageSpace.startedCopying();
SlotVisitor& visitor = m_slotVisitor;
HeapRootVisitor heapRootVisitor(visitor);
@@ -700,8 +706,10 @@
}
GCCOUNTER(VisitedValueCount, visitor.visitCount());
+ visitor.doneCopying();
visitor.reset();
m_sharedData.reset();
+ m_storageSpace.doneCopying();
m_operationInProgress = NoOperation;
}
@@ -822,11 +830,11 @@
// water mark to be proportional to the current size of the heap. The exact
// proportion is a bit arbitrary. A 2X multiplier gives a 1:1 (heap size :
// new bytes allocated) proportion, and seems to work well in benchmarks.
- size_t newSize = size();
+ size_t newSize = size() + m_storageSpace.totalMemoryUtilized();
size_t proportionalBytes = 2 * newSize;
if (fullGC) {
m_lastFullGCSize = newSize;
- m_objectSpace.setHighWaterMark(max(proportionalBytes, m_minBytesPerCycle));
+ setHighWaterMark(max(proportionalBytes, m_minBytesPerCycle));
}
JAVASCRIPTCORE_GC_END();
@@ -887,7 +895,7 @@
if (!m_numberOfFreeBlocks)
block = 0;
else {
- block = m_freeBlocks.removeHead();
+ block = static_cast<MarkedBlock*>(m_freeBlocks.removeHead());
ASSERT(block);
m_numberOfFreeBlocks--;
}
diff --git a/Source/JavaScriptCore/heap/Heap.h b/Source/JavaScriptCore/heap/Heap.h
index 3c78d62..44a6f86 100644
--- a/Source/JavaScriptCore/heap/Heap.h
+++ b/Source/JavaScriptCore/heap/Heap.h
@@ -31,12 +31,16 @@
#include "MarkedSpace.h"
#include "SlotVisitor.h"
#include "WriteBarrierSupport.h"
+#include <wtf/DoublyLinkedList.h>
#include <wtf/Forward.h>
#include <wtf/HashCountedSet.h>
#include <wtf/HashSet.h>
+#define COLLECT_ON_EVERY_ALLOCATION 0
+
namespace JSC {
+ class BumpSpace;
class CodeBlock;
class GCActivityCallback;
class GlobalCodeBlock;
@@ -57,7 +61,7 @@
typedef HashCountedSet<const char*> TypeCountSet;
enum OperationInProgress { NoOperation, Allocation, Collection };
-
+
// Heap size hint.
enum HeapSize { SmallHeap, LargeHeap };
@@ -65,6 +69,7 @@
WTF_MAKE_NONCOPYABLE(Heap);
public:
friend class JIT;
+ friend class MarkStackThreadSharedData;
static Heap* heap(JSValue); // 0 for immediate values
static Heap* heap(JSCell*);
@@ -92,6 +97,8 @@
MarkedSpace::SizeClass& sizeClassForObject(size_t bytes) { return m_objectSpace.sizeClassFor(bytes); }
void* allocate(size_t);
+ CheckedBoolean tryAllocateStorage(size_t, void**);
+ CheckedBoolean tryReallocateStorage(void**, size_t, size_t);
typedef void (*Finalizer)(JSCell*);
JS_EXPORT_PRIVATE void addFinalizer(JSCell*, Finalizer);
@@ -131,9 +138,14 @@
private:
friend class MarkedBlock;
friend class AllocationSpace;
+ friend class BumpSpace;
friend class SlotVisitor;
friend class CodeBlock;
+ size_t waterMark();
+ size_t highWaterMark();
+ void setHighWaterMark(size_t);
+
static const size_t minExtraCost = 256;
static const size_t maxExtraCost = 1024 * 1024;
@@ -175,11 +187,14 @@
const HeapSize m_heapSize;
const size_t m_minBytesPerCycle;
size_t m_lastFullGCSize;
+ size_t m_waterMark;
+ size_t m_highWaterMark;
OperationInProgress m_operationInProgress;
AllocationSpace m_objectSpace;
+ BumpSpace m_storageSpace;
- DoublyLinkedList<MarkedBlock> m_freeBlocks;
+ DoublyLinkedList<HeapBlock> m_freeBlocks;
size_t m_numberOfFreeBlocks;
ThreadIdentifier m_blockFreeingThread;
@@ -246,6 +261,21 @@
MarkedBlock::blockFor(cell)->setMarked(cell);
}
+ inline size_t Heap::waterMark()
+ {
+ return m_objectSpace.waterMark() + m_storageSpace.totalMemoryUtilized();
+ }
+
+ inline size_t Heap::highWaterMark()
+ {
+ return m_highWaterMark;
+ }
+
+ inline void Heap::setHighWaterMark(size_t newHighWaterMark)
+ {
+ m_highWaterMark = newHighWaterMark;
+ }
+
#if ENABLE(GGC)
inline uint8_t* Heap::addressOfCardFor(JSCell* cell)
{
@@ -308,6 +338,16 @@
ASSERT(isValidAllocation(bytes));
return m_objectSpace.allocate(bytes);
}
+
+ inline CheckedBoolean Heap::tryAllocateStorage(size_t bytes, void** outPtr)
+ {
+ return m_storageSpace.tryAllocate(bytes, outPtr);
+ }
+
+ inline CheckedBoolean Heap::tryReallocateStorage(void** ptr, size_t oldSize, size_t newSize)
+ {
+ return m_storageSpace.tryReallocate(ptr, oldSize, newSize);
+ }
} // namespace JSC
diff --git a/Source/JavaScriptCore/heap/HeapBlock.h b/Source/JavaScriptCore/heap/HeapBlock.h
new file mode 100644
index 0000000..1d66aad
--- /dev/null
+++ b/Source/JavaScriptCore/heap/HeapBlock.h
@@ -0,0 +1,52 @@
+/*
+ * Copyright (C) 2011 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef HeapBlock_h
+#define HeapBlock_h
+
+#include <wtf/DoublyLinkedList.h>
+#include <wtf/PageAllocationAligned.h>
+
+namespace JSC {
+
+class HeapBlock : public DoublyLinkedListNode<HeapBlock> {
+public:
+ HeapBlock(PageAllocationAligned& allocation)
+ : DoublyLinkedListNode<HeapBlock>()
+ , m_prev(0)
+ , m_next(0)
+ , m_allocation(allocation)
+ {
+ ASSERT(allocation);
+ }
+
+ HeapBlock* m_prev;
+ HeapBlock* m_next;
+ PageAllocationAligned m_allocation;
+};
+
+} // namespace JSC
+
+#endif
diff --git a/Source/JavaScriptCore/heap/MarkStack.cpp b/Source/JavaScriptCore/heap/MarkStack.cpp
index 02cf328..9a30923 100644
--- a/Source/JavaScriptCore/heap/MarkStack.cpp
+++ b/Source/JavaScriptCore/heap/MarkStack.cpp
@@ -26,6 +26,8 @@
#include "config.h"
#include "MarkStack.h"
+#include "BumpSpace.h"
+#include "BumpSpaceInlineMethods.h"
#include "ConservativeRoots.h"
#include "Heap.h"
#include "Options.h"
@@ -233,6 +235,7 @@
MarkStackThreadSharedData::MarkStackThreadSharedData(JSGlobalData* globalData)
: m_globalData(globalData)
+ , m_bumpSpace(&globalData->heap.m_storageSpace)
, m_sharedMarkStack(m_segmentAllocator)
, m_numberOfActiveParallelMarkers(0)
, m_parallelMarkersShouldExit(false)
@@ -337,7 +340,7 @@
void SlotVisitor::drain()
{
ASSERT(m_isInParallelMode);
-
+
#if ENABLE(PARALLEL_GC)
if (Options::numberOfGCMarkers > 1) {
while (!m_stack.isEmpty()) {
@@ -398,8 +401,11 @@
// for us to do.
while (true) {
// Did we reach termination?
- if (!m_shared.m_numberOfActiveParallelMarkers && m_shared.m_sharedMarkStack.isEmpty())
+ if (!m_shared.m_numberOfActiveParallelMarkers && m_shared.m_sharedMarkStack.isEmpty()) {
+ // Let any sleeping slaves know it's time for them to give their private BumpBlocks back
+ m_shared.m_markingCondition.broadcast();
return;
+ }
// Is there work to be done?
if (!m_shared.m_sharedMarkStack.isEmpty())
@@ -415,14 +421,19 @@
if (!m_shared.m_numberOfActiveParallelMarkers && m_shared.m_sharedMarkStack.isEmpty())
m_shared.m_markingCondition.broadcast();
- while (m_shared.m_sharedMarkStack.isEmpty() && !m_shared.m_parallelMarkersShouldExit)
+ while (m_shared.m_sharedMarkStack.isEmpty() && !m_shared.m_parallelMarkersShouldExit) {
+ if (!m_shared.m_numberOfActiveParallelMarkers && m_shared.m_sharedMarkStack.isEmpty())
+ doneCopying();
m_shared.m_markingCondition.wait(m_shared.m_markingLock);
+ }
// Is the VM exiting? If so, exit this thread.
- if (m_shared.m_parallelMarkersShouldExit)
+ if (m_shared.m_parallelMarkersShouldExit) {
+ doneCopying();
return;
+ }
}
-
+
m_stack.stealSomeCellsFrom(m_shared.m_sharedMarkStack);
m_shared.m_numberOfActiveParallelMarkers++;
}
@@ -445,6 +456,79 @@
m_opaqueRoots.clear();
}
+void SlotVisitor::startCopying()
+{
+ ASSERT(!m_copyBlock);
+ if (!m_shared.m_bumpSpace->borrowBlock(&m_copyBlock))
+ CRASH();
+}
+
+void* SlotVisitor::allocateNewSpace(void* ptr, size_t bytes)
+{
+ if (BumpSpace::isOversize(bytes)) {
+ m_shared.m_bumpSpace->pin(BumpSpace::oversizeBlockFor(ptr));
+ return 0;
+ }
+
+ if (m_shared.m_bumpSpace->isPinned(ptr))
+ return 0;
+
+ // The only time it's possible to have a null copy block is if we have just started copying.
+ if (!m_copyBlock)
+ startCopying();
+
+ if (!BumpSpace::fitsInBlock(m_copyBlock, bytes)) {
+ // We don't need to lock across these two calls because the master thread won't
+ // call doneCopying() because this thread is considered active.
+ m_shared.m_bumpSpace->doneFillingBlock(m_copyBlock);
+ if (!m_shared.m_bumpSpace->borrowBlock(&m_copyBlock))
+ CRASH();
+ }
+ return BumpSpace::allocateFromBlock(m_copyBlock, bytes);
+}
+
+void SlotVisitor::copy(void** ptr, size_t bytes)
+{
+ void* newPtr = 0;
+ if (!(newPtr = allocateNewSpace(*ptr, bytes)))
+ return;
+
+ memcpy(newPtr, *ptr, bytes);
+ *ptr = newPtr;
+}
+
+void SlotVisitor::copyAndAppend(void** ptr, size_t bytes, JSValue* values, unsigned length)
+{
+ void* oldPtr = *ptr;
+ void* newPtr = allocateNewSpace(oldPtr, bytes);
+ if (newPtr) {
+ size_t jsValuesOffset = static_cast<size_t>(reinterpret_cast<char*>(values) - static_cast<char*>(oldPtr));
+
+ JSValue* newValues = reinterpret_cast<JSValue*>(static_cast<char*>(newPtr) + jsValuesOffset);
+ for (unsigned i = 0; i < length; i++) {
+ JSValue& value = values[i];
+ newValues[i] = value;
+ if (!value)
+ continue;
+ internalAppend(value);
+ }
+
+ memcpy(newPtr, oldPtr, jsValuesOffset);
+ *ptr = newPtr;
+ } else
+ append(values, length);
+}
+
+void SlotVisitor::doneCopying()
+{
+ if (!m_copyBlock)
+ return;
+
+ m_shared.m_bumpSpace->doneFillingBlock(m_copyBlock);
+
+ m_copyBlock = 0;
+}
+
void SlotVisitor::harvestWeakReferences()
{
for (WeakReferenceHarvester* current = m_shared.m_weakReferenceHarvesters.head(); current; current = current->next())
diff --git a/Source/JavaScriptCore/heap/MarkStack.h b/Source/JavaScriptCore/heap/MarkStack.h
index 9562090..6923cdd 100644
--- a/Source/JavaScriptCore/heap/MarkStack.h
+++ b/Source/JavaScriptCore/heap/MarkStack.h
@@ -26,6 +26,7 @@
#ifndef MarkStack_h
#define MarkStack_h
+#include "BumpSpace.h"
#include "HandleTypes.h"
#include "Options.h"
#include "JSValue.h"
@@ -181,6 +182,7 @@
#endif
JSGlobalData* m_globalData;
+ BumpSpace* m_bumpSpace;
MarkStackSegmentAllocator m_segmentAllocator;
diff --git a/Source/JavaScriptCore/heap/MarkedBlock.cpp b/Source/JavaScriptCore/heap/MarkedBlock.cpp
index 771c9c0..715f25d 100644
--- a/Source/JavaScriptCore/heap/MarkedBlock.cpp
+++ b/Source/JavaScriptCore/heap/MarkedBlock.cpp
@@ -40,9 +40,9 @@
return new (NotNull, allocation.base()) MarkedBlock(allocation, heap, cellSize);
}
-MarkedBlock* MarkedBlock::recycle(MarkedBlock* block, size_t cellSize)
+MarkedBlock* MarkedBlock::recycle(MarkedBlock* block, Heap* heap, size_t cellSize)
{
- return new (NotNull, block) MarkedBlock(block->m_allocation, block->m_heap, cellSize);
+ return new (NotNull, block) MarkedBlock(block->m_allocation, heap, cellSize);
}
void MarkedBlock::destroy(MarkedBlock* block)
@@ -50,13 +50,14 @@
block->m_allocation.deallocate();
}
-MarkedBlock::MarkedBlock(const PageAllocationAligned& allocation, Heap* heap, size_t cellSize)
- : m_atomsPerCell((cellSize + atomSize - 1) / atomSize)
+MarkedBlock::MarkedBlock(PageAllocationAligned& allocation, Heap* heap, size_t cellSize)
+ : HeapBlock(allocation)
+ , m_atomsPerCell((cellSize + atomSize - 1) / atomSize)
, m_endAtom(atomsPerBlock - m_atomsPerCell + 1)
, m_state(New) // All cells start out unmarked.
- , m_allocation(allocation)
, m_heap(heap)
{
+ ASSERT(heap);
HEAP_LOG_BLOCK_STATE_TRANSITION(this);
}
diff --git a/Source/JavaScriptCore/heap/MarkedBlock.h b/Source/JavaScriptCore/heap/MarkedBlock.h
index 8c665dd..00eb54b 100644
--- a/Source/JavaScriptCore/heap/MarkedBlock.h
+++ b/Source/JavaScriptCore/heap/MarkedBlock.h
@@ -23,6 +23,7 @@
#define MarkedBlock_h
#include "CardSet.h"
+#include "HeapBlock.h"
#include <wtf/Bitmap.h>
#include <wtf/DoublyLinkedList.h>
@@ -50,7 +51,6 @@
typedef uintptr_t Bits;
- static const size_t KB = 1024;
static const size_t MB = 1024 * 1024;
bool isZapped(const JSCell*);
@@ -63,14 +63,14 @@
// size is equal to the difference between the cell size and the object
// size.
- class MarkedBlock : public DoublyLinkedListNode<MarkedBlock> {
+ class MarkedBlock : public HeapBlock {
friend class WTF::DoublyLinkedListNode<MarkedBlock>;
public:
// Ensure natural alignment for native types whilst recognizing that the smallest
// object the heap will commonly allocate is four words.
static const size_t atomSize = 4 * sizeof(void*);
static const size_t atomShift = 5;
- static const size_t blockSize = 16 * KB;
+ static const size_t blockSize = 64 * KB;
static const size_t blockMask = ~(blockSize - 1); // blockSize must be a power of two.
static const size_t atomsPerBlock = blockSize / atomSize; // ~0.4% overhead
@@ -90,7 +90,7 @@
};
static MarkedBlock* create(Heap*, size_t cellSize);
- static MarkedBlock* recycle(MarkedBlock*, size_t cellSize);
+ static MarkedBlock* recycle(MarkedBlock*, Heap*, size_t cellSize);
static void destroy(MarkedBlock*);
static bool isAtomAligned(const void*);
@@ -162,7 +162,7 @@
typedef char Atom[atomSize];
- MarkedBlock(const PageAllocationAligned&, Heap*, size_t cellSize);
+ MarkedBlock(PageAllocationAligned&, Heap*, size_t cellSize);
Atom* atoms();
size_t atomNumber(const void*);
void callDestructor(JSCell*);
@@ -180,10 +180,7 @@
WTF::Bitmap<atomsPerBlock, WTF::BitmapNotAtomic> m_marks;
#endif
BlockState m_state;
- PageAllocationAligned m_allocation;
Heap* m_heap;
- MarkedBlock* m_prev;
- MarkedBlock* m_next;
};
inline size_t MarkedBlock::firstAtom()
diff --git a/Source/JavaScriptCore/heap/MarkedSpace.cpp b/Source/JavaScriptCore/heap/MarkedSpace.cpp
index acbd8ac..c12e9ba 100644
--- a/Source/JavaScriptCore/heap/MarkedSpace.cpp
+++ b/Source/JavaScriptCore/heap/MarkedSpace.cpp
@@ -33,7 +33,6 @@
MarkedSpace::MarkedSpace(Heap* heap)
: m_waterMark(0)
, m_nurseryWaterMark(0)
- , m_highWaterMark(0)
, m_heap(heap)
{
for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep)
diff --git a/Source/JavaScriptCore/heap/MarkedSpace.h b/Source/JavaScriptCore/heap/MarkedSpace.h
index 751fe2f..3b2c828 100644
--- a/Source/JavaScriptCore/heap/MarkedSpace.h
+++ b/Source/JavaScriptCore/heap/MarkedSpace.h
@@ -54,7 +54,7 @@
MarkedBlock::FreeCell* firstFreeCell;
MarkedBlock* currentBlock;
- DoublyLinkedList<MarkedBlock> blockList;
+ DoublyLinkedList<HeapBlock> blockList;
size_t cellSize;
};
@@ -71,9 +71,7 @@
void canonicalizeCellLivenessData();
size_t waterMark();
- size_t highWaterMark();
size_t nurseryWaterMark();
- void setHighWaterMark(size_t);
template<typename Functor> typename Functor::ReturnType forEachBlock(Functor&); // Safe to remove the current item while iterating.
template<typename Functor> typename Functor::ReturnType forEachBlock();
@@ -93,7 +91,6 @@
FixedArray<SizeClass, impreciseCount> m_impreciseSizeClasses;
size_t m_waterMark;
size_t m_nurseryWaterMark;
- size_t m_highWaterMark;
Heap* m_heap;
};
@@ -102,21 +99,11 @@
return m_waterMark;
}
-inline size_t MarkedSpace::highWaterMark()
-{
- return m_highWaterMark;
-}
-
inline size_t MarkedSpace::nurseryWaterMark()
{
return m_nurseryWaterMark;
}
-inline void MarkedSpace::setHighWaterMark(size_t highWaterMark)
-{
- m_highWaterMark = highWaterMark;
-}
-
inline MarkedSpace::SizeClass& MarkedSpace::sizeClassFor(size_t bytes)
{
ASSERT(bytes && bytes <= maxCellSize);
@@ -129,7 +116,7 @@
{
MarkedBlock::FreeCell* firstFreeCell = sizeClass.firstFreeCell;
if (!firstFreeCell) {
- for (MarkedBlock*& block = sizeClass.currentBlock; block; block = block->next()) {
+ for (MarkedBlock*& block = sizeClass.currentBlock; block; block = static_cast<MarkedBlock*>(block->next())) {
firstFreeCell = block->sweep(MarkedBlock::SweepToFreeList);
if (firstFreeCell)
break;
@@ -152,19 +139,19 @@
{
for (size_t i = 0; i < preciseCount; ++i) {
SizeClass& sizeClass = m_preciseSizeClasses[i];
- MarkedBlock* next;
- for (MarkedBlock* block = sizeClass.blockList.head(); block; block = next) {
+ HeapBlock* next;
+ for (HeapBlock* block = sizeClass.blockList.head(); block; block = next) {
next = block->next();
- functor(block);
+ functor(static_cast<MarkedBlock*>(block));
}
}
for (size_t i = 0; i < impreciseCount; ++i) {
SizeClass& sizeClass = m_impreciseSizeClasses[i];
- MarkedBlock* next;
- for (MarkedBlock* block = sizeClass.blockList.head(); block; block = next) {
+ HeapBlock* next;
+ for (HeapBlock* block = sizeClass.blockList.head(); block; block = next) {
next = block->next();
- functor(block);
+ functor(static_cast<MarkedBlock*>(block));
}
}
@@ -186,7 +173,7 @@
inline void MarkedSpace::SizeClass::resetAllocator()
{
- currentBlock = blockList.head();
+ currentBlock = static_cast<MarkedBlock*>(blockList.head());
}
inline void MarkedSpace::SizeClass::zapFreeList()
diff --git a/Source/JavaScriptCore/heap/SlotVisitor.h b/Source/JavaScriptCore/heap/SlotVisitor.h
index 142d8ca..e49a9a6 100644
--- a/Source/JavaScriptCore/heap/SlotVisitor.h
+++ b/Source/JavaScriptCore/heap/SlotVisitor.h
@@ -26,6 +26,7 @@
#ifndef SlotVisitor_h
#define SlotVisitor_h
+#include "BumpSpace.h"
#include "MarkStack.h"
namespace JSC {
@@ -59,8 +60,15 @@
void harvestWeakReferences();
void finalizeUnconditionalFinalizers();
+
+ void startCopying();
+ void copy(void**, size_t);
+ void copyAndAppend(void**, size_t, JSValue*, unsigned);
+ void doneCopying();
private:
+ void* allocateNewSpace(void*, size_t);
+
void donateSlow();
void donateKnownParallel()
@@ -69,10 +77,13 @@
return;
donateSlow();
}
+
+ BumpBlock* m_copyBlock;
};
inline SlotVisitor::SlotVisitor(MarkStackThreadSharedData& shared)
: MarkStack(shared)
+ , m_copyBlock(0)
{
}
diff --git a/Source/JavaScriptCore/heap/TinyBloomFilter.h b/Source/JavaScriptCore/heap/TinyBloomFilter.h
index 82b5863..a75ce8c 100644
--- a/Source/JavaScriptCore/heap/TinyBloomFilter.h
+++ b/Source/JavaScriptCore/heap/TinyBloomFilter.h
@@ -36,6 +36,7 @@
void add(Bits);
bool ruleOut(Bits) const; // True for 0.
+ void reset();
private:
Bits m_bits;
@@ -62,6 +63,11 @@
return false;
}
+inline void TinyBloomFilter::reset()
+{
+ m_bits = 0;
+}
+
} // namespace JSC
#endif // TinyBloomFilter_h
diff --git a/Source/JavaScriptCore/runtime/JSArray.cpp b/Source/JavaScriptCore/runtime/JSArray.cpp
index bd6e79a..fa658a6 100644
--- a/Source/JavaScriptCore/runtime/JSArray.cpp
+++ b/Source/JavaScriptCore/runtime/JSArray.cpp
@@ -24,6 +24,8 @@
#include "JSArray.h"
#include "ArrayPrototype.h"
+#include "BumpSpace.h"
+#include "BumpSpaceInlineMethods.h"
#include "CachedCall.h"
#include "Error.h"
#include "Executable.h"
@@ -130,7 +132,10 @@
JSArray::JSArray(JSGlobalData& globalData, Structure* structure)
: JSNonFinalObject(globalData, structure)
+ , m_indexBias(0)
, m_storage(0)
+ , m_sparseValueMap(0)
+ , m_subclassData(0)
{
}
@@ -142,13 +147,14 @@
unsigned initialVectorLength = BASE_VECTOR_LEN;
unsigned initialStorageSize = storageSize(initialVectorLength);
- m_storage = static_cast<ArrayStorage*>(fastMalloc(initialStorageSize));
+ void* newStorage = 0;
+ if (!globalData.heap.tryAllocateStorage(initialStorageSize, &newStorage))
+ CRASH();
+
+ m_storage = static_cast<ArrayStorage*>(newStorage);
m_storage->m_allocBase = m_storage;
m_storage->m_length = initialLength;
- m_indexBias = 0;
m_vectorLength = initialVectorLength;
- m_storage->m_sparseValueMap = 0;
- m_storage->subclassData = 0;
m_storage->m_numValuesInVector = 0;
#if CHECK_ARRAY_CONSISTENCY
m_storage->m_inCompactInitialization = false;
@@ -159,8 +165,6 @@
vector[i].clear();
checkConsistency();
-
- Heap::heap(this)->reportExtraMemoryCost(initialStorageSize);
}
JSArray* JSArray::tryFinishCreationUninitialized(JSGlobalData& globalData, unsigned initialLength)
@@ -175,14 +179,16 @@
unsigned initialVectorLength = max(initialLength, BASE_VECTOR_LEN);
unsigned initialStorageSize = storageSize(initialVectorLength);
- m_storage = static_cast<ArrayStorage*>(fastMalloc(initialStorageSize));
+ void* newStorage = 0;
+ if (!globalData.heap.tryAllocateStorage(initialStorageSize, &newStorage))
+ CRASH();
+
+ m_storage = static_cast<ArrayStorage*>(newStorage);
m_storage->m_allocBase = m_storage;
m_storage->m_length = 0;
- m_indexBias = 0;
m_vectorLength = initialVectorLength;
- m_storage->m_sparseValueMap = 0;
- m_storage->subclassData = 0;
m_storage->m_numValuesInVector = initialLength;
+
#if CHECK_ARRAY_CONSISTENCY
m_storage->m_inCompactInitialization = true;
#endif
@@ -191,21 +197,14 @@
for (size_t i = initialLength; i < initialVectorLength; ++i)
vector[i].clear();
- Heap::heap(this)->reportExtraMemoryCost(initialStorageSize);
return this;
}
JSArray::~JSArray()
{
ASSERT(jsCast<JSArray*>(this));
-
- // If we are unable to allocate memory for m_storage then this may be null.
- if (!m_storage)
- return;
-
checkConsistency(DestructorConsistencyCheck);
- delete m_storage->m_sparseValueMap;
- fastFree(m_storage->m_allocBase);
+ delete m_sparseValueMap;
}
void JSArray::destroy(JSCell* cell)
@@ -313,10 +312,10 @@
void JSArray::enterSparseMode(JSGlobalData& globalData)
{
ArrayStorage* storage = m_storage;
- SparseArrayValueMap* map = storage->m_sparseValueMap;
+ SparseArrayValueMap* map = m_sparseValueMap;
if (!map)
- map = storage->m_sparseValueMap = new SparseArrayValueMap;
+ map = m_sparseValueMap = new SparseArrayValueMap;
if (map->sparseMode())
return;
@@ -332,10 +331,13 @@
map->add(this, i).first->second.set(globalData, this, value);
}
- ArrayStorage* newStorage = static_cast<ArrayStorage*>(fastMalloc(storageSize(0)));
+ void* newRawStorage = 0;
+ if (!globalData.heap.tryAllocateStorage(storageSize(0), &newRawStorage))
+ CRASH();
+
+ ArrayStorage* newStorage = static_cast<ArrayStorage*>(newRawStorage);
memcpy(newStorage, m_storage, storageSize(0));
newStorage->m_allocBase = newStorage;
- fastFree(m_storage);
m_storage = newStorage;
m_indexBias = 0;
m_vectorLength = 0;
@@ -405,7 +407,7 @@
enterSparseMode(exec->globalData());
}
- SparseArrayValueMap* map = m_storage->m_sparseValueMap;
+ SparseArrayValueMap* map = m_sparseValueMap;
ASSERT(map);
// 1. Let current be the result of calling the [[GetOwnProperty]] internal method of O with property name P.
@@ -514,7 +516,7 @@
enterSparseMode(exec->globalData());
- SparseArrayValueMap* map = m_storage->m_sparseValueMap;
+ SparseArrayValueMap* map = m_sparseValueMap;
ASSERT(map);
map->setLengthIsReadOnly();
}
@@ -626,7 +628,7 @@
{
JSArray* thisObject = jsCast<JSArray*>(cell);
ArrayStorage* storage = thisObject->m_storage;
-
+
if (i >= storage->m_length) {
if (i > MAX_ARRAY_INDEX)
return thisObject->methodTable()->getOwnPropertySlot(thisObject, exec, Identifier::from(exec, i), slot);
@@ -639,7 +641,7 @@
slot.setValue(value);
return true;
}
- } else if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ } else if (SparseArrayValueMap* map = thisObject->m_sparseValueMap) {
SparseArrayValueMap::iterator it = map->find(i);
if (it != map->notFound()) {
it->second.get(slot);
@@ -687,7 +689,7 @@
descriptor.setDescriptor(value.get(), 0);
return true;
}
- } else if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ } else if (SparseArrayValueMap* map = thisObject->m_sparseValueMap) {
SparseArrayValueMap::iterator it = map->find(i);
if (it != map->notFound()) {
it->second.get(descriptor);
@@ -768,7 +770,7 @@
ASSERT(i <= MAX_ARRAY_INDEX);
ArrayStorage* storage = m_storage;
- SparseArrayValueMap* map = storage->m_sparseValueMap;
+ SparseArrayValueMap* map = m_sparseValueMap;
// First, handle cases where we don't currently have a sparse map.
if (LIKELY(!map)) {
@@ -777,7 +779,7 @@
storage->m_length = i + 1;
// Check that it is sensible to still be using a vector, and then try to grow the vector.
- if (LIKELY((isDenseEnoughForVector(i, storage->m_numValuesInVector)) && increaseVectorLength(i + 1))) {
+ if (LIKELY((isDenseEnoughForVector(i, storage->m_numValuesInVector)) && increaseVectorLength(globalData, i + 1))) {
// success! - reread m_storage since it has likely been reallocated, and store to the vector.
storage = m_storage;
storage->m_vector[i].set(globalData, this, value);
@@ -786,7 +788,7 @@
}
// We don't want to, or can't use a vector to hold this property - allocate a sparse map & add the value.
map = new SparseArrayValueMap;
- storage->m_sparseValueMap = map;
+ m_sparseValueMap = map;
map->put(exec, this, i, value);
return;
}
@@ -806,7 +808,7 @@
// We are currently using a map - check whether we still want to be doing so.
// We will continue to use a sparse map if SparseMode is set, a vector would be too sparse, or if allocation fails.
unsigned numValuesInArray = storage->m_numValuesInVector + map->size();
- if (map->sparseMode() || !isDenseEnoughForVector(length, numValuesInArray) || !increaseVectorLength(length)) {
+ if (map->sparseMode() || !isDenseEnoughForVector(length, numValuesInArray) || !increaseVectorLength(exec->globalData(), length)) {
map->put(exec, this, i, value);
return;
}
@@ -821,7 +823,7 @@
for (SparseArrayValueMap::const_iterator it = map->begin(); it != end; ++it)
vector[it->first].set(globalData, this, it->second.getNonSparseMode());
delete map;
- storage->m_sparseValueMap = 0;
+ m_sparseValueMap = 0;
// Store the new property into the vector.
WriteBarrier<Unknown>& valueSlot = vector[i];
@@ -860,7 +862,7 @@
valueSlot.clear();
--storage->m_numValuesInVector;
}
- } else if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ } else if (SparseArrayValueMap* map = thisObject->m_sparseValueMap) {
SparseArrayValueMap::iterator it = map->find(i);
if (it != map->notFound()) {
if (it->second.attributes & DontDelete)
@@ -895,7 +897,7 @@
propertyNames.add(Identifier::from(exec, i));
}
- if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ if (SparseArrayValueMap* map = thisObject->m_sparseValueMap) {
Vector<unsigned> keys;
keys.reserveCapacity(map->size());
@@ -943,7 +945,7 @@
return min(increasedLength, MAX_STORAGE_VECTOR_LENGTH);
}
-bool JSArray::increaseVectorLength(unsigned newLength)
+bool JSArray::increaseVectorLength(JSGlobalData& globalData, unsigned newLength)
{
// This function leaves the array in an internally inconsistent state, because it does not move any values from sparse value map
// to the vector. Callers have to account for that, because they can do it more efficiently.
@@ -955,15 +957,16 @@
unsigned vectorLength = m_vectorLength;
ASSERT(newLength > vectorLength);
unsigned newVectorLength = getNewVectorLength(newLength);
- void* baseStorage = storage->m_allocBase;
// Fast case - there is no precapacity. In these cases a realloc makes sense.
if (LIKELY(!m_indexBias)) {
- if (!tryFastRealloc(baseStorage, storageSize(newVectorLength)).getValue(baseStorage))
+ void* newStorage = storage->m_allocBase;
+ if (!globalData.heap.tryReallocateStorage(&newStorage, storageSize(vectorLength), storageSize(newVectorLength)))
return false;
- storage = m_storage = reinterpret_cast_ptr<ArrayStorage*>(baseStorage);
- m_storage->m_allocBase = baseStorage;
+ storage = m_storage = reinterpret_cast_ptr<ArrayStorage*>(static_cast<char*>(newStorage));
+ m_storage->m_allocBase = newStorage;
+ ASSERT(m_storage->m_allocBase);
WriteBarrier<Unknown>* vector = storage->m_vector;
for (unsigned i = vectorLength; i < newVectorLength; ++i)
@@ -971,7 +974,6 @@
m_vectorLength = newVectorLength;
- Heap::heap(this)->reportExtraMemoryCost(storageSize(newVectorLength) - storageSize(vectorLength));
return true;
}
@@ -979,15 +981,11 @@
unsigned newIndexBias = min(m_indexBias >> 1, MAX_STORAGE_VECTOR_LENGTH - newVectorLength);
// Calculate new stoarge capcity, allowing room for the pre-capacity.
unsigned newStorageCapacity = newVectorLength + newIndexBias;
- void* newAllocBase;
- if (!tryFastMalloc(storageSize(newStorageCapacity)).getValue(newAllocBase))
+ void* newAllocBase = 0;
+ if (!globalData.heap.tryAllocateStorage(storageSize(newStorageCapacity), &newAllocBase))
return false;
// The sum of m_vectorLength and m_indexBias will never exceed MAX_STORAGE_VECTOR_LENGTH.
ASSERT(m_vectorLength <= MAX_STORAGE_VECTOR_LENGTH && (MAX_STORAGE_VECTOR_LENGTH - m_vectorLength) >= m_indexBias);
- unsigned currentCapacity = m_vectorLength + m_indexBias;
- // Currently there is no way to report to the heap that the extra capacity is shrinking!
- if (newStorageCapacity > currentCapacity)
- Heap::heap(this)->reportExtraMemoryCost((newStorageCapacity - currentCapacity) * sizeof(WriteBarrier<Unknown>));
m_vectorLength = newVectorLength;
m_indexBias = newIndexBias;
@@ -999,14 +997,13 @@
m_storage->m_vector[i].clear();
// Free the old allocation, update m_allocBase.
- fastFree(m_storage->m_allocBase);
m_storage->m_allocBase = newAllocBase;
return true;
}
// This method makes room in the vector, but leaves the new space uncleared.
-bool JSArray::unshiftCountSlowCase(unsigned count)
+bool JSArray::unshiftCountSlowCase(JSGlobalData& globalData, unsigned count)
{
// If not, we should have handled this on the fast path.
ASSERT(count > m_indexBias);
@@ -1037,19 +1034,16 @@
// Step 2:
// We're either going to choose to allocate a new ArrayStorage, or we're going to reuse the existing on.
- void* newAllocBase;
+ void* newAllocBase = 0;
unsigned newStorageCapacity;
// If the current storage array is sufficiently large (but not too large!) then just keep using it.
if (currentCapacity > desiredCapacity && isDenseEnoughForVector(currentCapacity, requiredVectorLength)) {
newAllocBase = storage->m_allocBase;
newStorageCapacity = currentCapacity;
} else {
- if (!tryFastMalloc(storageSize(desiredCapacity)).getValue(newAllocBase))
+ if (!globalData.heap.tryAllocateStorage(storageSize(desiredCapacity), &newAllocBase))
return false;
newStorageCapacity = desiredCapacity;
- // Currently there is no way to report to the heap that the extra capacity is shrinking!
- if (desiredCapacity > currentCapacity)
- Heap::heap(this)->reportExtraMemoryCost((desiredCapacity - currentCapacity) * sizeof(WriteBarrier<Unknown>));
}
// Step 3:
@@ -1082,7 +1076,6 @@
// Are we copying into a new allocation?
if (newAllocBase != m_storage->m_allocBase) {
// Free the old allocation, update m_allocBase.
- fastFree(m_storage->m_allocBase);
m_storage->m_allocBase = newAllocBase;
// We need to clear any entries in the vector beyond length. We only need to
@@ -1104,9 +1097,9 @@
unsigned length = storage->m_length;
// If the length is read only then we enter sparse mode, so should enter the following 'if'.
- ASSERT(isLengthWritable() || storage->m_sparseValueMap);
+ ASSERT(isLengthWritable() || m_sparseValueMap);
- if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ if (SparseArrayValueMap* map = m_sparseValueMap) {
// Fail if the length is not writable.
if (map->lengthIsReadOnly())
return reject(exec, throwException, StrictModeReadonlyPropertyWriteError);
@@ -1143,7 +1136,7 @@
map->remove(keys[i]);
if (map->isEmpty()) {
delete map;
- storage->m_sparseValueMap = 0;
+ m_sparseValueMap = 0;
}
}
}
@@ -1193,7 +1186,7 @@
result = jsUndefined();
} else {
result = jsUndefined();
- if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ if (SparseArrayValueMap* map = m_sparseValueMap) {
SparseArrayValueMap::iterator it = map->find(length);
if (it != map->notFound()) {
unsigned attributes = it->second.attributes;
@@ -1211,7 +1204,7 @@
map->remove(it);
if (map->isEmpty() && !map->sparseMode()) {
delete map;
- storage->m_sparseValueMap = 0;
+ m_sparseValueMap = 0;
}
}
}
@@ -1334,7 +1327,7 @@
memmove(newBaseStorage, storage, storageSize(0));
m_storage = reinterpret_cast_ptr<ArrayStorage*>(newBaseStorage);
m_vectorLength += count;
- } else if (!unshiftCountSlowCase(count)) {
+ } else if (!unshiftCountSlowCase(exec->globalData(), count)) {
throwOutOfMemoryError(exec);
return;
}
@@ -1352,13 +1345,21 @@
ASSERT(thisObject->structure()->typeInfo().overridesVisitChildren());
JSNonFinalObject::visitChildren(thisObject, visitor);
-
- ArrayStorage* storage = thisObject->m_storage;
- unsigned usedVectorLength = std::min(storage->m_length, thisObject->m_vectorLength);
- visitor.appendValues(storage->m_vector, usedVectorLength);
+ if (thisObject->m_storage) {
+ ArrayStorage* storage = thisObject->m_storage;
+ void* baseStorage = storage->m_allocBase;
- if (SparseArrayValueMap* map = storage->m_sparseValueMap)
+ visitor.copyAndAppend(reinterpret_cast<void**>(&baseStorage), storageSize(thisObject->m_vectorLength + thisObject->m_indexBias), storage->m_vector->slot(), thisObject->m_vectorLength);
+
+ if (baseStorage != thisObject->m_storage->m_allocBase) {
+ thisObject->m_storage = reinterpret_cast<ArrayStorage*>(static_cast<char*>(baseStorage) + sizeof(JSValue) * thisObject->m_indexBias);
+ thisObject->m_storage->m_allocBase = baseStorage;
+ ASSERT(thisObject->m_storage->m_allocBase);
+ }
+ }
+
+ if (SparseArrayValueMap* map = thisObject->m_sparseValueMap)
map->visitChildren(visitor);
}
@@ -1382,8 +1383,8 @@
ArrayStorage* storage = m_storage;
- unsigned lengthNotIncludingUndefined = compactForSorting();
- if (storage->m_sparseValueMap) {
+ unsigned lengthNotIncludingUndefined = compactForSorting(exec->globalData());
+ if (m_sparseValueMap) {
throwOutOfMemoryError(exec);
return;
}
@@ -1415,10 +1416,8 @@
{
ASSERT(!inSparseMode());
- ArrayStorage* storage = m_storage;
-
- unsigned lengthNotIncludingUndefined = compactForSorting();
- if (storage->m_sparseValueMap) {
+ unsigned lengthNotIncludingUndefined = compactForSorting(exec->globalData());
+ if (m_sparseValueMap) {
throwOutOfMemoryError(exec);
return;
}
@@ -1440,7 +1439,7 @@
Heap::heap(this)->pushTempSortVector(&values);
for (size_t i = 0; i < lengthNotIncludingUndefined; i++) {
- JSValue value = storage->m_vector[i].get();
+ JSValue value = m_storage->m_vector[i].get();
ASSERT(!value.isUndefined());
values[i].first = value;
}
@@ -1470,13 +1469,13 @@
// If the toString function changed the length of the array or vector storage,
// increase the length to handle the orignal number of actual values.
if (m_vectorLength < lengthNotIncludingUndefined)
- increaseVectorLength(lengthNotIncludingUndefined);
- if (storage->m_length < lengthNotIncludingUndefined)
- storage->m_length = lengthNotIncludingUndefined;
+ increaseVectorLength(exec->globalData(), lengthNotIncludingUndefined);
+ if (m_storage->m_length < lengthNotIncludingUndefined)
+ m_storage->m_length = lengthNotIncludingUndefined;
JSGlobalData& globalData = exec->globalData();
for (size_t i = 0; i < lengthNotIncludingUndefined; i++)
- storage->m_vector[i].set(globalData, this, values[i].first);
+ m_storage->m_vector[i].set(globalData, this, values[i].first);
Heap::heap(this)->popTempSortVector(&values);
@@ -1566,18 +1565,16 @@
checkConsistency();
- ArrayStorage* storage = m_storage;
-
// FIXME: This ignores exceptions raised in the compare function or in toNumber.
// The maximum tree depth is compiled in - but the caller is clearly up to no good
// if a larger array is passed.
- ASSERT(storage->m_length <= static_cast<unsigned>(std::numeric_limits<int>::max()));
- if (storage->m_length > static_cast<unsigned>(std::numeric_limits<int>::max()))
+ ASSERT(m_storage->m_length <= static_cast<unsigned>(std::numeric_limits<int>::max()));
+ if (m_storage->m_length > static_cast<unsigned>(std::numeric_limits<int>::max()))
return;
- unsigned usedVectorLength = min(storage->m_length, m_vectorLength);
- unsigned nodeCount = usedVectorLength + (storage->m_sparseValueMap ? storage->m_sparseValueMap->size() : 0);
+ unsigned usedVectorLength = min(m_storage->m_length, m_vectorLength);
+ unsigned nodeCount = usedVectorLength + (m_sparseValueMap ? m_sparseValueMap->size() : 0);
if (!nodeCount)
return;
@@ -1605,14 +1602,14 @@
// Iterate over the array, ignoring missing values, counting undefined ones, and inserting all other ones into the tree.
for (; numDefined < usedVectorLength; ++numDefined) {
- JSValue v = storage->m_vector[numDefined].get();
+ JSValue v = m_storage->m_vector[numDefined].get();
if (!v || v.isUndefined())
break;
tree.abstractor().m_nodes[numDefined].value = v;
tree.insert(numDefined);
}
for (unsigned i = numDefined; i < usedVectorLength; ++i) {
- JSValue v = storage->m_vector[i].get();
+ JSValue v = m_storage->m_vector[i].get();
if (v) {
if (v.isUndefined())
++numUndefined;
@@ -1626,17 +1623,15 @@
unsigned newUsedVectorLength = numDefined + numUndefined;
- if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ if (SparseArrayValueMap* map = m_sparseValueMap) {
newUsedVectorLength += map->size();
if (newUsedVectorLength > m_vectorLength) {
// Check that it is possible to allocate an array large enough to hold all the entries.
- if ((newUsedVectorLength > MAX_STORAGE_VECTOR_LENGTH) || !increaseVectorLength(newUsedVectorLength)) {
+ if ((newUsedVectorLength > MAX_STORAGE_VECTOR_LENGTH) || !increaseVectorLength(exec->globalData(), newUsedVectorLength)) {
throwOutOfMemoryError(exec);
return;
}
}
-
- storage = m_storage;
SparseArrayValueMap::const_iterator end = map->end();
for (SparseArrayValueMap::const_iterator it = map->begin(); it != end; ++it) {
@@ -1646,7 +1641,7 @@
}
delete map;
- storage->m_sparseValueMap = 0;
+ m_sparseValueMap = 0;
}
ASSERT(tree.abstractor().m_nodes.size() >= numDefined);
@@ -1659,19 +1654,19 @@
iter.start_iter_least(tree);
JSGlobalData& globalData = exec->globalData();
for (unsigned i = 0; i < numDefined; ++i) {
- storage->m_vector[i].set(globalData, this, tree.abstractor().m_nodes[*iter].value);
+ m_storage->m_vector[i].set(globalData, this, tree.abstractor().m_nodes[*iter].value);
++iter;
}
// Put undefined values back in.
for (unsigned i = numDefined; i < newUsedVectorLength; ++i)
- storage->m_vector[i].setUndefined();
+ m_storage->m_vector[i].setUndefined();
// Ensure that unused values in the vector are zeroed out.
for (unsigned i = newUsedVectorLength; i < usedVectorLength; ++i)
- storage->m_vector[i].clear();
+ m_storage->m_vector[i].clear();
- storage->m_numValuesInVector = newUsedVectorLength;
+ m_storage->m_numValuesInVector = newUsedVectorLength;
checkConsistency(SortConsistencyCheck);
}
@@ -1712,7 +1707,7 @@
callFrame->setArgument(i, get(exec, i));
}
-unsigned JSArray::compactForSorting()
+unsigned JSArray::compactForSorting(JSGlobalData& globalData)
{
ASSERT(!inSparseMode());
@@ -1743,12 +1738,12 @@
unsigned newUsedVectorLength = numDefined + numUndefined;
- if (SparseArrayValueMap* map = storage->m_sparseValueMap) {
+ if (SparseArrayValueMap* map = m_sparseValueMap) {
newUsedVectorLength += map->size();
if (newUsedVectorLength > m_vectorLength) {
// Check that it is possible to allocate an array large enough to hold all the entries - if not,
// exception is thrown by caller.
- if ((newUsedVectorLength > MAX_STORAGE_VECTOR_LENGTH) || !increaseVectorLength(newUsedVectorLength))
+ if ((newUsedVectorLength > MAX_STORAGE_VECTOR_LENGTH) || !increaseVectorLength(globalData, newUsedVectorLength))
return 0;
storage = m_storage;
@@ -1759,7 +1754,7 @@
storage->m_vector[numDefined++].setWithoutWriteBarrier(it->second.getNonSparseMode());
delete map;
- storage->m_sparseValueMap = 0;
+ m_sparseValueMap = 0;
}
for (unsigned i = numDefined; i < newUsedVectorLength; ++i)
@@ -1776,12 +1771,12 @@
void* JSArray::subclassData() const
{
- return m_storage->subclassData;
+ return m_subclassData;
}
void JSArray::setSubclassData(void* d)
{
- m_storage->subclassData = d;
+ m_subclassData = d;
}
#if CHECK_ARRAY_CONSISTENCY
@@ -1794,7 +1789,7 @@
ASSERT(storage);
if (type == SortConsistencyCheck)
- ASSERT(!storage->m_sparseValueMap);
+ ASSERT(!m_sparseValueMap);
unsigned numValuesInVector = 0;
for (unsigned i = 0; i < m_vectorLength; ++i) {
@@ -1811,9 +1806,9 @@
ASSERT(numValuesInVector == storage->m_numValuesInVector);
ASSERT(numValuesInVector <= storage->m_length);
- if (storage->m_sparseValueMap) {
- SparseArrayValueMap::iterator end = storage->m_sparseValueMap->end();
- for (SparseArrayValueMap::iterator it = storage->m_sparseValueMap->begin(); it != end; ++it) {
+ if (m_sparseValueMap) {
+ SparseArrayValueMap::iterator end = m_sparseValueMap->end();
+ for (SparseArrayValueMap::iterator it = m_sparseValueMap->begin(); it != end; ++it) {
unsigned index = it->first;
ASSERT(index < storage->m_length);
ASSERT(index >= storage->m_vectorLength);
diff --git a/Source/JavaScriptCore/runtime/JSArray.h b/Source/JavaScriptCore/runtime/JSArray.h
index 284c977..5e47977 100644
--- a/Source/JavaScriptCore/runtime/JSArray.h
+++ b/Source/JavaScriptCore/runtime/JSArray.h
@@ -114,8 +114,6 @@
struct ArrayStorage {
unsigned m_length; // The "length" property on the array
unsigned m_numValuesInVector;
- SparseArrayValueMap* m_sparseValueMap;
- void* subclassData; // A JSArray subclass can use this to fill the vector lazily.
void* m_allocBase; // Pointer to base address returned by malloc(). Keeping this pointer does eliminate false positives from the leak detector.
#if CHECK_ARRAY_CONSISTENCY
bool m_inCompactInitialization;
@@ -232,7 +230,7 @@
bool inSparseMode()
{
- SparseArrayValueMap* map = m_storage->m_sparseValueMap;
+ SparseArrayValueMap* map = m_sparseValueMap;
return map && map->sparseMode();
}
@@ -270,7 +268,7 @@
private:
bool isLengthWritable()
{
- SparseArrayValueMap* map = m_storage->m_sparseValueMap;
+ SparseArrayValueMap* map = m_sparseValueMap;
return !map || !map->lengthIsReadOnly();
}
@@ -283,10 +281,10 @@
void putByIndexBeyondVectorLength(ExecState*, unsigned propertyName, JSValue);
unsigned getNewVectorLength(unsigned desiredLength);
- bool increaseVectorLength(unsigned newLength);
- bool unshiftCountSlowCase(unsigned count);
+ bool increaseVectorLength(JSGlobalData&, unsigned newLength);
+ bool unshiftCountSlowCase(JSGlobalData&, unsigned count);
- unsigned compactForSorting();
+ unsigned compactForSorting(JSGlobalData&);
enum ConsistencyCheckType { NormalConsistencyCheck, DestructorConsistencyCheck, SortConsistencyCheck };
void checkConsistency(ConsistencyCheckType = NormalConsistencyCheck);
@@ -294,6 +292,10 @@
unsigned m_vectorLength; // The valid length of m_vector
unsigned m_indexBias; // The number of JSValue sized blocks before ArrayStorage.
ArrayStorage *m_storage;
+
+ // FIXME: Maybe SparseArrayValueMap should be put into its own JSCell?
+ SparseArrayValueMap* m_sparseValueMap;
+ void* m_subclassData; // A JSArray subclass can use this to fill the vector lazily.
};
JSArray* asArray(JSValue);
diff --git a/Source/JavaScriptCore/wtf/CheckedBoolean.h b/Source/JavaScriptCore/wtf/CheckedBoolean.h
new file mode 100644
index 0000000..c65c70e
--- /dev/null
+++ b/Source/JavaScriptCore/wtf/CheckedBoolean.h
@@ -0,0 +1,61 @@
+/*
+ * Copyright (C) 2011 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef CheckedBoolean_h
+#define CheckedBoolean_h
+
+#include <wtf/Assertions.h>
+
+class CheckedBoolean {
+public:
+ CheckedBoolean(bool value)
+ : m_value(value)
+#if !ASSERT_DISABLED
+ , m_checked(false)
+#endif
+ {
+ }
+
+ ~CheckedBoolean()
+ {
+ ASSERT(m_checked);
+ }
+
+ operator bool()
+ {
+#if !ASSERT_DISABLED
+ m_checked = true;
+#endif
+ return m_value;
+ }
+
+private:
+ bool m_value;
+#if !ASSERT_DISABLED
+ bool m_checked;
+#endif
+};
+
+#endif
diff --git a/Source/JavaScriptCore/wtf/DoublyLinkedList.h b/Source/JavaScriptCore/wtf/DoublyLinkedList.h
index 18aa00e..361d71d 100644
--- a/Source/JavaScriptCore/wtf/DoublyLinkedList.h
+++ b/Source/JavaScriptCore/wtf/DoublyLinkedList.h
@@ -77,6 +77,7 @@
T* head() const;
T* removeHead();
+ void push(T*);
void append(T*);
void remove(T*);
@@ -115,6 +116,24 @@
return m_head;
}
+template<typename T> inline void DoublyLinkedList<T>::push(T* node)
+{
+ if (!m_head) {
+ ASSERT(!m_tail);
+ m_head = node;
+ m_tail = node;
+ node->setPrev(0);
+ node->setNext(0);
+ return;
+ }
+
+ ASSERT(m_tail);
+ m_head->setPrev(node);
+ node->setNext(m_head);
+ node->setPrev(0);
+ m_head = node;
+}
+
template<typename T> inline void DoublyLinkedList<T>::append(T* node)
{
if (!m_tail) {
diff --git a/Source/JavaScriptCore/wtf/StdLibExtras.h b/Source/JavaScriptCore/wtf/StdLibExtras.h
index 3f99c4d..e4d7c8f 100644
--- a/Source/JavaScriptCore/wtf/StdLibExtras.h
+++ b/Source/JavaScriptCore/wtf/StdLibExtras.h
@@ -107,6 +107,13 @@
namespace WTF {
+static const size_t KB = 1024;
+
+inline bool isPointerAligned(void* p)
+{
+ return !((intptr_t)(p) & (sizeof(char*) - 1));
+}
+
/*
* C++'s idea of a reinterpret_cast lacks sufficient cojones.
*/
@@ -283,6 +290,8 @@
return location;
}
+using WTF::KB;
+using WTF::isPointerAligned;
using WTF::binarySearch;
using WTF::bitwise_cast;
using WTF::safeCast;
