We're collecting pathologically due to small allocations
https://bugs.webkit.org/show_bug.cgi?id=84404
Reviewed by Geoffrey Garen.
No change in performance on run-jsc-benchmarks.
* dfg/DFGSpeculativeJIT.h: Replacing m_firstFreeCell with m_freeList.
(JSC::DFG::SpeculativeJIT::emitAllocateBasicJSObject):
* heap/CopiedSpace.cpp: Getting rid of any water mark related stuff, since it's no
longer useful.
(JSC::CopiedSpace::CopiedSpace):
(JSC::CopiedSpace::tryAllocateSlowCase): We now only call didAllocate here rather than
carrying out a somewhat complicated accounting job for our old water mark throughout CopiedSpace.
(JSC::CopiedSpace::tryAllocateOversize): Call the new didAllocate to notify the Heap of
newly allocated stuff.
(JSC::CopiedSpace::tryReallocateOversize):
(JSC::CopiedSpace::doneFillingBlock):
(JSC::CopiedSpace::doneCopying):
(JSC::CopiedSpace::destroy):
* heap/CopiedSpace.h:
(CopiedSpace):
* heap/CopiedSpaceInlineMethods.h:
(JSC::CopiedSpace::startedCopying):
* heap/Heap.cpp: Removed water mark related stuff, replaced with new bytesAllocated and
bytesAllocatedLimit to track how much memory has been allocated since the last collection.
(JSC::Heap::Heap):
(JSC::Heap::reportExtraMemoryCostSlowCase):
(JSC::Heap::collect): We now set the new limit of bytes that we can allocate before triggering
a collection to be the size of the Heap after the previous collection. Thus, we still have our
2x allocation amount.
(JSC::Heap::didAllocate): Notifies the GC activity timer of how many bytes have been allocated
thus far and then adds the new number of bytes to the current total.
(JSC):
* heap/Heap.h: Removed water mark related stuff.
(JSC::Heap::notifyIsSafeToCollect):
(Heap):
(JSC::Heap::shouldCollect):
(JSC):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateHelper): Refactored to use MarkedBlock's new FreeList struct.
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::addBlock):
* heap/MarkedAllocator.h:
(MarkedAllocator):
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::zapFreeList): Refactored to take in a FreeList instead of a FreeCell.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::specializedSweep):
(JSC::MarkedBlock::sweep):
(JSC::MarkedBlock::sweepHelper):
(JSC::MarkedBlock::zapFreeList):
* heap/MarkedBlock.h:
(FreeList): Added a new struct that keeps track of the current MarkedAllocator's
free list including the number of bytes of stuff in the free list so that when the free list is
exhausted, the correct amount can be reported to Heap.
(MarkedBlock):
(JSC::MarkedBlock::FreeList::FreeList):
(JSC):
* heap/MarkedSpace.cpp: Removing all water mark related stuff.
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::resetAllocators):
* heap/MarkedSpace.h:
(MarkedSpace):
(JSC):
* heap/WeakSet.cpp:
(JSC::WeakSet::findAllocator): Refactored to use the didAllocate interface with the Heap. This
function still needs work though now that the Heap knows how many bytes have been allocated
since the last collection.
* jit/JITInlineMethods.h: Refactored to use MarkedBlock's new FreeList struct.
(JSC::JIT::emitAllocateBasicJSObject): Ditto.
* llint/LowLevelInterpreter.asm: Ditto.
* runtime/GCActivityCallback.cpp:
(JSC::DefaultGCActivityCallback::didAllocate):
* runtime/GCActivityCallback.h:
(JSC::GCActivityCallback::didAllocate): Renamed willAllocate to didAllocate to indicate that
the allocation that is being reported has already taken place.
(DefaultGCActivityCallback):
* runtime/GCActivityCallbackCF.cpp:
(JSC):
(JSC::DefaultGCActivityCallback::didAllocate): Refactored to return early if the amount of
allocation since the last collection is not above a threshold (initially arbitrarily chosen to
be 128KB).
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@114698 268f45cc-cd09-0410-ab3c-d52691b4dbfc
diff --git a/Source/JavaScriptCore/ChangeLog b/Source/JavaScriptCore/ChangeLog
index 46893c3..b18870f 100644
--- a/Source/JavaScriptCore/ChangeLog
+++ b/Source/JavaScriptCore/ChangeLog
@@ -1,3 +1,90 @@
+2012-04-19 Mark Hahnenberg <mhahnenberg@apple.com>
+
+ We're collecting pathologically due to small allocations
+ https://bugs.webkit.org/show_bug.cgi?id=84404
+
+ Reviewed by Geoffrey Garen.
+
+ No change in performance on run-jsc-benchmarks.
+
+ * dfg/DFGSpeculativeJIT.h: Replacing m_firstFreeCell with m_freeList.
+ (JSC::DFG::SpeculativeJIT::emitAllocateBasicJSObject):
+ * heap/CopiedSpace.cpp: Getting rid of any water mark related stuff, since it's no
+ longer useful.
+ (JSC::CopiedSpace::CopiedSpace):
+ (JSC::CopiedSpace::tryAllocateSlowCase): We now only call didAllocate here rather than
+ carrying out a somewhat complicated accounting job for our old water mark throughout CopiedSpace.
+ (JSC::CopiedSpace::tryAllocateOversize): Call the new didAllocate to notify the Heap of
+ newly allocated stuff.
+ (JSC::CopiedSpace::tryReallocateOversize):
+ (JSC::CopiedSpace::doneFillingBlock):
+ (JSC::CopiedSpace::doneCopying):
+ (JSC::CopiedSpace::destroy):
+ * heap/CopiedSpace.h:
+ (CopiedSpace):
+ * heap/CopiedSpaceInlineMethods.h:
+ (JSC::CopiedSpace::startedCopying):
+ * heap/Heap.cpp: Removed water mark related stuff, replaced with new bytesAllocated and
+ bytesAllocatedLimit to track how much memory has been allocated since the last collection.
+ (JSC::Heap::Heap):
+ (JSC::Heap::reportExtraMemoryCostSlowCase):
+ (JSC::Heap::collect): We now set the new limit of bytes that we can allocate before triggering
+ a collection to be the size of the Heap after the previous collection. Thus, we still have our
+ 2x allocation amount.
+ (JSC::Heap::didAllocate): Notifies the GC activity timer of how many bytes have been allocated
+ thus far and then adds the new number of bytes to the current total.
+ (JSC):
+ * heap/Heap.h: Removed water mark related stuff.
+ (JSC::Heap::notifyIsSafeToCollect):
+ (Heap):
+ (JSC::Heap::shouldCollect):
+ (JSC):
+ * heap/MarkedAllocator.cpp:
+ (JSC::MarkedAllocator::tryAllocateHelper): Refactored to use MarkedBlock's new FreeList struct.
+ (JSC::MarkedAllocator::allocateSlowCase):
+ (JSC::MarkedAllocator::addBlock):
+ * heap/MarkedAllocator.h:
+ (MarkedAllocator):
+ (JSC::MarkedAllocator::MarkedAllocator):
+ (JSC::MarkedAllocator::allocate):
+ (JSC::MarkedAllocator::zapFreeList): Refactored to take in a FreeList instead of a FreeCell.
+ * heap/MarkedBlock.cpp:
+ (JSC::MarkedBlock::specializedSweep):
+ (JSC::MarkedBlock::sweep):
+ (JSC::MarkedBlock::sweepHelper):
+ (JSC::MarkedBlock::zapFreeList):
+ * heap/MarkedBlock.h:
+ (FreeList): Added a new struct that keeps track of the current MarkedAllocator's
+ free list including the number of bytes of stuff in the free list so that when the free list is
+ exhausted, the correct amount can be reported to Heap.
+ (MarkedBlock):
+ (JSC::MarkedBlock::FreeList::FreeList):
+ (JSC):
+ * heap/MarkedSpace.cpp: Removing all water mark related stuff.
+ (JSC::MarkedSpace::MarkedSpace):
+ (JSC::MarkedSpace::resetAllocators):
+ * heap/MarkedSpace.h:
+ (MarkedSpace):
+ (JSC):
+ * heap/WeakSet.cpp:
+ (JSC::WeakSet::findAllocator): Refactored to use the didAllocate interface with the Heap. This
+ function still needs work though now that the Heap knows how many bytes have been allocated
+ since the last collection.
+ * jit/JITInlineMethods.h: Refactored to use MarkedBlock's new FreeList struct.
+ (JSC::JIT::emitAllocateBasicJSObject): Ditto.
+ * llint/LowLevelInterpreter.asm: Ditto.
+ * runtime/GCActivityCallback.cpp:
+ (JSC::DefaultGCActivityCallback::didAllocate):
+ * runtime/GCActivityCallback.h:
+ (JSC::GCActivityCallback::didAllocate): Renamed willAllocate to didAllocate to indicate that
+ the allocation that is being reported has already taken place.
+ (DefaultGCActivityCallback):
+ * runtime/GCActivityCallbackCF.cpp:
+ (JSC):
+ (JSC::DefaultGCActivityCallback::didAllocate): Refactored to return early if the amount of
+ allocation since the last collection is not above a threshold (initially arbitrarily chosen to
+ be 128KB).
+
2012-04-19 Filip Pizlo <fpizlo@apple.com>
MacroAssemblerARMv7::branchTruncateDoubleToUint32 should obey the overflow signal
diff --git a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
index fb7be1a..60524a9 100644
--- a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
+++ b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
@@ -1794,7 +1794,7 @@
else
allocator = &m_jit.globalData()->heap.allocatorForObjectWithoutDestructor(sizeof(ClassType));
- m_jit.loadPtr(&allocator->m_firstFreeCell, resultGPR);
+ m_jit.loadPtr(&allocator->m_freeList.head, resultGPR);
slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, resultGPR));
// The object is half-allocated: we have what we know is a fresh object, but
@@ -1806,7 +1806,7 @@
// Now that we have scratchGPR back, remove the object from the free list
m_jit.loadPtr(MacroAssembler::Address(resultGPR), scratchGPR);
- m_jit.storePtr(scratchGPR, &allocator->m_firstFreeCell);
+ m_jit.storePtr(scratchGPR, &allocator->m_freeList.head);
// Initialize the object's classInfo pointer
m_jit.storePtr(MacroAssembler::TrustedImmPtr(&ClassType::s_info), MacroAssembler::Address(resultGPR, JSCell::classInfoOffset()));
diff --git a/Source/JavaScriptCore/heap/CopiedSpace.cpp b/Source/JavaScriptCore/heap/CopiedSpace.cpp
index a6e4250..f69a2a8 100644
--- a/Source/JavaScriptCore/heap/CopiedSpace.cpp
+++ b/Source/JavaScriptCore/heap/CopiedSpace.cpp
@@ -37,7 +37,6 @@
, m_fromSpace(0)
, m_inCopyingPhase(false)
, m_numberOfLoanedBlocks(0)
- , m_waterMark(0)
{
}
@@ -52,12 +51,10 @@
CheckedBoolean CopiedSpace::tryAllocateSlowCase(size_t bytes, void** outPtr)
{
- m_heap->activityCallback()->willAllocate();
-
if (isOversize(bytes))
return tryAllocateOversize(bytes, outPtr);
- m_waterMark += m_allocator.currentCapacity();
+ m_heap->didAllocate(m_allocator.currentCapacity());
if (!addNewBlock()) {
*outPtr = 0;
@@ -86,7 +83,7 @@
*outPtr = allocateFromBlock(block, bytes);
- m_waterMark += block->capacity();
+ m_heap->didAllocate(blockSize);
return true;
}
@@ -138,7 +135,6 @@
if (isOversize(oldSize)) {
CopiedBlock* oldBlock = oversizeBlockFor(oldPtr);
m_oversizeBlocks.remove(oldBlock);
- m_waterMark -= oldBlock->capacity();
oldBlock->m_allocation.deallocate();
}
@@ -165,11 +161,6 @@
}
{
- MutexLocker locker(m_memoryStatsLock);
- m_waterMark += block->capacity();
- }
-
- {
MutexLocker locker(m_loanedBlocksLock);
ASSERT(m_numberOfLoanedBlocks > 0);
m_numberOfLoanedBlocks--;
@@ -193,7 +184,6 @@
if (block->m_isPinned) {
block->m_isPinned = false;
m_toSpace->push(block);
- m_waterMark += block->capacity();
continue;
}
@@ -211,10 +201,8 @@
if (!curr->m_isPinned) {
m_oversizeBlocks.remove(curr);
curr->m_allocation.deallocate();
- } else {
+ } else
curr->m_isPinned = false;
- m_waterMark += curr->capacity();
- }
curr = next;
}
@@ -281,8 +269,6 @@
CopiedBlock* block = static_cast<CopiedBlock*>(m_oversizeBlocks.removeHead());
block->m_allocation.deallocate();
}
-
- m_waterMark = 0;
}
size_t CopiedSpace::size()
diff --git a/Source/JavaScriptCore/heap/CopiedSpace.h b/Source/JavaScriptCore/heap/CopiedSpace.h
index cc54bce..cf815b6 100644
--- a/Source/JavaScriptCore/heap/CopiedSpace.h
+++ b/Source/JavaScriptCore/heap/CopiedSpace.h
@@ -66,7 +66,6 @@
bool contains(void*, CopiedBlock*&);
- size_t waterMark() { return m_waterMark; }
size_t size();
size_t capacity();
@@ -92,8 +91,6 @@
static bool fitsInBlock(CopiedBlock*, size_t);
static CopiedBlock* oversizeBlockFor(void* ptr);
- size_t calculateWaterMark();
-
Heap* m_heap;
CopiedAllocator m_allocator;
@@ -117,9 +114,6 @@
ThreadCondition m_loanedBlocksCondition;
size_t m_numberOfLoanedBlocks;
- Mutex m_memoryStatsLock;
- size_t m_waterMark;
-
static const size_t s_maxAllocationSize = 32 * KB;
static const size_t s_initialBlockNum = 16;
static const size_t s_blockMask = ~(HeapBlock::s_blockSize - 1);
diff --git a/Source/JavaScriptCore/heap/CopiedSpaceInlineMethods.h b/Source/JavaScriptCore/heap/CopiedSpaceInlineMethods.h
index 9fec1fd..fc8081f 100644
--- a/Source/JavaScriptCore/heap/CopiedSpaceInlineMethods.h
+++ b/Source/JavaScriptCore/heap/CopiedSpaceInlineMethods.h
@@ -56,8 +56,6 @@
m_toSpaceFilter.reset();
m_allocator.startedCopying();
- m_waterMark = 0;
-
ASSERT(!m_inCopyingPhase);
ASSERT(!m_numberOfLoanedBlocks);
m_inCopyingPhase = true;
diff --git a/Source/JavaScriptCore/heap/Heap.cpp b/Source/JavaScriptCore/heap/Heap.cpp
index d40fe90..33fa39b 100644
--- a/Source/JavaScriptCore/heap/Heap.cpp
+++ b/Source/JavaScriptCore/heap/Heap.cpp
@@ -313,7 +313,8 @@
: m_heapSize(heapSize)
, m_minBytesPerCycle(heapSizeForHint(heapSize))
, m_lastFullGCSize(0)
- , m_highWaterMark(m_minBytesPerCycle)
+ , m_bytesAllocatedLimit(m_minBytesPerCycle)
+ , m_bytesAllocated(0)
, m_operationInProgress(NoOperation)
, m_objectSpace(this)
, m_storageSpace(this)
@@ -465,7 +466,9 @@
// if a large value survives one garbage collection, there is not much point to
// collecting more frequently as long as it stays alive.
- addToWaterMark(cost);
+ didAllocate(cost);
+ if (shouldCollect())
+ collect(DoNotSweep);
}
void Heap::protect(JSValue k)
@@ -848,16 +851,17 @@
shrink();
}
- // To avoid pathological GC churn in large heaps, we set the allocation high
- // water mark to be proportional to the current size of the heap. The exact
- // proportion is a bit arbitrary. A 2X multiplier gives a 1:1 (heap size :
+ // To avoid pathological GC churn in large heaps, we set the new allocation
+ // limit to be the current size of the heap. This heuristic
+ // is a bit arbitrary. Using the current size of the heap after this
+ // collection gives us a 2X multiplier, which is a 1:1 (heap size :
// new bytes allocated) proportion, and seems to work well in benchmarks.
size_t newSize = size();
- size_t proportionalBytes = 2 * newSize;
if (fullGC) {
m_lastFullGCSize = newSize;
- m_highWaterMark = max(proportionalBytes, m_minBytesPerCycle);
+ m_bytesAllocatedLimit = max(newSize, m_minBytesPerCycle);
}
+ m_bytesAllocated = 0;
double lastGCEndTime = WTF::currentTime();
m_lastGCLength = lastGCEndTime - lastGCStartTime;
JAVASCRIPTCORE_GC_END();
@@ -886,6 +890,12 @@
return m_activityCallback.get();
}
+void Heap::didAllocate(size_t bytes)
+{
+ m_activityCallback->didAllocate(m_bytesAllocated);
+ m_bytesAllocated += bytes;
+}
+
bool Heap::isValidAllocation(size_t bytes)
{
if (!isValidThreadState(m_globalData))
diff --git a/Source/JavaScriptCore/heap/Heap.h b/Source/JavaScriptCore/heap/Heap.h
index df2af07..1f850f6 100644
--- a/Source/JavaScriptCore/heap/Heap.h
+++ b/Source/JavaScriptCore/heap/Heap.h
@@ -112,8 +112,11 @@
void removeFunctionExecutable(FunctionExecutable*);
void notifyIsSafeToCollect() { m_isSafeToCollect = true; }
- JS_EXPORT_PRIVATE void collectAllGarbage();
+ JS_EXPORT_PRIVATE void collectAllGarbage();
+ enum SweepToggle { DoNotSweep, DoSweep };
+ bool shouldCollect();
+ void collect(SweepToggle);
void reportExtraMemoryCost(size_t cost);
JS_EXPORT_PRIVATE void protect(JSValue);
@@ -144,12 +147,12 @@
void getConservativeRegisterRoots(HashSet<JSCell*>& roots);
- void addToWaterMark(size_t);
-
double lastGCLength() { return m_lastGCLength; }
JS_EXPORT_PRIVATE void discardAllCompiledCode();
+ void didAllocate(size_t);
+
private:
friend class CodeBlock;
friend class LLIntOffsetsExtractor;
@@ -163,10 +166,6 @@
void* allocateWithDestructor(size_t);
void* allocateWithoutDestructor(size_t);
- size_t waterMark();
- size_t highWaterMark();
- bool shouldCollect();
-
static const size_t minExtraCost = 256;
static const size_t maxExtraCost = 1024 * 1024;
@@ -192,8 +191,6 @@
void harvestWeakReferences();
void finalizeUnconditionalFinalizers();
- enum SweepToggle { DoNotSweep, DoSweep };
- void collect(SweepToggle);
void shrink();
void releaseFreeBlocks();
void sweep();
@@ -208,7 +205,9 @@
const HeapSize m_heapSize;
const size_t m_minBytesPerCycle;
size_t m_lastFullGCSize;
- size_t m_highWaterMark;
+
+ size_t m_bytesAllocatedLimit;
+ size_t m_bytesAllocated;
OperationInProgress m_operationInProgress;
MarkedSpace m_objectSpace;
@@ -257,7 +256,7 @@
#if ENABLE(GGC)
return m_objectSpace.nurseryWaterMark() >= m_minBytesPerCycle && m_isSafeToCollect;
#else
- return waterMark() >= highWaterMark() && m_isSafeToCollect;
+ return m_bytesAllocated > m_bytesAllocatedLimit && m_isSafeToCollect;
#endif
}
@@ -293,23 +292,6 @@
MarkedBlock::blockFor(cell)->setMarked(cell);
}
- inline size_t Heap::waterMark()
- {
- return m_objectSpace.waterMark() + m_storageSpace.waterMark();
- }
-
- inline size_t Heap::highWaterMark()
- {
- return m_highWaterMark;
- }
-
- inline void Heap::addToWaterMark(size_t size)
- {
- m_objectSpace.addToWaterMark(size);
- if (waterMark() > highWaterMark())
- collect(DoNotSweep);
- }
-
#if ENABLE(GGC)
inline uint8_t* Heap::addressOfCardFor(JSCell* cell)
{
diff --git a/Source/JavaScriptCore/heap/MarkedAllocator.cpp b/Source/JavaScriptCore/heap/MarkedAllocator.cpp
index 044b642..bf0c052 100644
--- a/Source/JavaScriptCore/heap/MarkedAllocator.cpp
+++ b/Source/JavaScriptCore/heap/MarkedAllocator.cpp
@@ -8,23 +8,22 @@
inline void* MarkedAllocator::tryAllocateHelper()
{
- MarkedBlock::FreeCell* firstFreeCell = m_firstFreeCell;
- if (!firstFreeCell) {
+ if (!m_freeList.head) {
for (MarkedBlock*& block = m_currentBlock; block; block = static_cast<MarkedBlock*>(block->next())) {
- firstFreeCell = block->sweep(MarkedBlock::SweepToFreeList);
- if (firstFreeCell)
+ m_freeList = block->sweep(MarkedBlock::SweepToFreeList);
+ if (m_freeList.head)
break;
- m_markedSpace->didConsumeFreeList(block);
block->didConsumeFreeList();
}
- if (!firstFreeCell)
+ if (!m_freeList.head)
return 0;
}
- ASSERT(firstFreeCell);
- m_firstFreeCell = firstFreeCell->next;
- return firstFreeCell;
+ MarkedBlock::FreeCell* head = m_freeList.head;
+ m_freeList.head = head->next;
+ ASSERT(head);
+ return head;
}
inline void* MarkedAllocator::tryAllocate()
@@ -42,7 +41,8 @@
ASSERT(m_heap->m_operationInProgress == NoOperation);
#endif
- m_heap->activityCallback()->willAllocate();
+ ASSERT(!m_freeList.head);
+ m_heap->didAllocate(m_freeList.bytes);
void* result = tryAllocate();
@@ -71,7 +71,7 @@
if (result)
return result;
- ASSERT(m_heap->waterMark() < m_heap->highWaterMark());
+ ASSERT(!m_heap->shouldCollect());
addBlock(allocateBlock(AllocationMustSucceed));
@@ -108,11 +108,11 @@
void MarkedAllocator::addBlock(MarkedBlock* block)
{
ASSERT(!m_currentBlock);
- ASSERT(!m_firstFreeCell);
+ ASSERT(!m_freeList.head);
m_blockList.append(block);
m_currentBlock = block;
- m_firstFreeCell = block->sweep(MarkedBlock::SweepToFreeList);
+ m_freeList = block->sweep(MarkedBlock::SweepToFreeList);
}
void MarkedAllocator::removeBlock(MarkedBlock* block)
diff --git a/Source/JavaScriptCore/heap/MarkedAllocator.h b/Source/JavaScriptCore/heap/MarkedAllocator.h
index 1c6af77..e2f4af0 100644
--- a/Source/JavaScriptCore/heap/MarkedAllocator.h
+++ b/Source/JavaScriptCore/heap/MarkedAllocator.h
@@ -41,7 +41,7 @@
void* tryAllocateHelper();
MarkedBlock* allocateBlock(AllocationEffort);
- MarkedBlock::FreeCell* m_firstFreeCell;
+ MarkedBlock::FreeList m_freeList;
MarkedBlock* m_currentBlock;
DoublyLinkedList<HeapBlock> m_blockList;
size_t m_cellSize;
@@ -51,8 +51,7 @@
};
inline MarkedAllocator::MarkedAllocator()
- : m_firstFreeCell(0)
- , m_currentBlock(0)
+ : m_currentBlock(0)
, m_cellSize(0)
, m_cellsNeedDestruction(true)
, m_heap(0)
@@ -70,13 +69,13 @@
inline void* MarkedAllocator::allocate()
{
- MarkedBlock::FreeCell* firstFreeCell = m_firstFreeCell;
+ MarkedBlock::FreeCell* head = m_freeList.head;
// This is a light-weight fast path to cover the most common case.
- if (UNLIKELY(!firstFreeCell))
+ if (UNLIKELY(!head))
return allocateSlowCase();
- m_firstFreeCell = firstFreeCell->next;
- return firstFreeCell;
+ m_freeList.head = head->next;
+ return head;
}
inline void MarkedAllocator::reset()
@@ -87,12 +86,12 @@
inline void MarkedAllocator::zapFreeList()
{
if (!m_currentBlock) {
- ASSERT(!m_firstFreeCell);
+ ASSERT(!m_freeList.head);
return;
}
- m_currentBlock->zapFreeList(m_firstFreeCell);
- m_firstFreeCell = 0;
+ m_currentBlock->zapFreeList(m_freeList);
+ m_freeList.head = 0;
}
template <typename Functor> inline void MarkedAllocator::forEachBlock(Functor& functor)
diff --git a/Source/JavaScriptCore/heap/MarkedBlock.cpp b/Source/JavaScriptCore/heap/MarkedBlock.cpp
index 75c21e7..3a58b5a 100644
--- a/Source/JavaScriptCore/heap/MarkedBlock.cpp
+++ b/Source/JavaScriptCore/heap/MarkedBlock.cpp
@@ -77,7 +77,7 @@
}
template<MarkedBlock::BlockState blockState, MarkedBlock::SweepMode sweepMode, bool destructorCallNeeded>
-MarkedBlock::FreeCell* MarkedBlock::specializedSweep()
+MarkedBlock::FreeList MarkedBlock::specializedSweep()
{
ASSERT(blockState != Allocated && blockState != FreeListed);
ASSERT(destructorCallNeeded || sweepMode != SweepOnly);
@@ -86,6 +86,7 @@
// This is fine, since the allocation code makes no assumptions about the
// order of the free list.
FreeCell* head = 0;
+ size_t count = 0;
for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
if (blockState == Marked && m_marks.get(i))
continue;
@@ -101,19 +102,20 @@
FreeCell* freeCell = reinterpret_cast<FreeCell*>(cell);
freeCell->next = head;
head = freeCell;
+ ++count;
}
}
m_state = ((sweepMode == SweepToFreeList) ? FreeListed : Zapped);
- return head;
+ return FreeList(head, count * cellSize());
}
-MarkedBlock::FreeCell* MarkedBlock::sweep(SweepMode sweepMode)
+MarkedBlock::FreeList MarkedBlock::sweep(SweepMode sweepMode)
{
HEAP_LOG_BLOCK_STATE_TRANSITION(this);
if (sweepMode == SweepOnly && !m_cellsNeedDestruction)
- return 0;
+ return FreeList();
if (m_cellsNeedDestruction)
return sweepHelper<true>(sweepMode);
@@ -121,7 +123,7 @@
}
template<bool destructorCallNeeded>
-MarkedBlock::FreeCell* MarkedBlock::sweepHelper(SweepMode sweepMode)
+MarkedBlock::FreeList MarkedBlock::sweepHelper(SweepMode sweepMode)
{
switch (m_state) {
case New:
@@ -130,10 +132,10 @@
case FreeListed:
// Happens when a block transitions to fully allocated.
ASSERT(sweepMode == SweepToFreeList);
- return 0;
+ return FreeList();
case Allocated:
ASSERT_NOT_REACHED();
- return 0;
+ return FreeList();
case Marked:
return sweepMode == SweepToFreeList
? specializedSweep<Marked, SweepToFreeList, destructorCallNeeded>()
@@ -145,12 +147,13 @@
}
ASSERT_NOT_REACHED();
- return 0;
+ return FreeList();
}
-void MarkedBlock::zapFreeList(FreeCell* firstFreeCell)
+void MarkedBlock::zapFreeList(const FreeList& freeList)
{
HEAP_LOG_BLOCK_STATE_TRANSITION(this);
+ FreeCell* head = freeList.head;
if (m_state == Marked) {
// If the block is in the Marked state then we know that:
@@ -159,7 +162,7 @@
// fact that their mark bits are unset.
// Hence if the block is Marked we need to leave it Marked.
- ASSERT(!firstFreeCell);
+ ASSERT(!head);
return;
}
@@ -176,7 +179,7 @@
// dead objects will have 0 in their vtables and live objects will have
// non-zero vtables, which is consistent with the block being zapped.
- ASSERT(!firstFreeCell);
+ ASSERT(!head);
return;
}
@@ -188,7 +191,7 @@
// way to tell what's live vs dead. We use zapping for that.
FreeCell* next;
- for (FreeCell* current = firstFreeCell; current; current = next) {
+ for (FreeCell* current = head; current; current = next) {
next = current->next;
reinterpret_cast<JSCell*>(current)->zap();
}
diff --git a/Source/JavaScriptCore/heap/MarkedBlock.h b/Source/JavaScriptCore/heap/MarkedBlock.h
index 3d0182e..429b7c0 100644
--- a/Source/JavaScriptCore/heap/MarkedBlock.h
+++ b/Source/JavaScriptCore/heap/MarkedBlock.h
@@ -87,6 +87,14 @@
FreeCell* next;
};
+ struct FreeList {
+ FreeCell* head;
+ size_t bytes;
+
+ FreeList();
+ FreeList(FreeCell*, size_t);
+ };
+
struct VoidFunctor {
typedef void ReturnType;
void returnValue() { }
@@ -105,13 +113,13 @@
void* allocate();
enum SweepMode { SweepOnly, SweepToFreeList };
- FreeCell* sweep(SweepMode = SweepOnly);
+ FreeList sweep(SweepMode = SweepOnly);
// While allocating from a free list, MarkedBlock temporarily has bogus
// cell liveness data. To restore accurate cell liveness data, call one
// of these functions:
void didConsumeFreeList(); // Call this once you've allocated all the items in the free list.
- void zapFreeList(FreeCell* firstFreeCell); // Call this to undo the free list.
+ void zapFreeList(const FreeList&); // Call this to undo the free list.
void clearMarks();
size_t markCount();
@@ -163,7 +171,7 @@
static const size_t atomAlignmentMask = atomSize - 1; // atomSize must be a power of two.
enum BlockState { New, FreeListed, Allocated, Marked, Zapped };
- template<bool destructorCallNeeded> FreeCell* sweepHelper(SweepMode = SweepOnly);
+ template<bool destructorCallNeeded> FreeList sweepHelper(SweepMode = SweepOnly);
typedef char Atom[atomSize];
@@ -171,7 +179,7 @@
Atom* atoms();
size_t atomNumber(const void*);
void callDestructor(JSCell*);
- template<BlockState, SweepMode, bool destructorCallNeeded> FreeCell* specializedSweep();
+ template<BlockState, SweepMode, bool destructorCallNeeded> FreeList specializedSweep();
#if ENABLE(GGC)
CardSet<bytesPerCard, blockSize> m_cards;
@@ -189,6 +197,18 @@
Heap* m_heap;
};
+ inline MarkedBlock::FreeList::FreeList()
+ : head(0)
+ , bytes(0)
+ {
+ }
+
+ inline MarkedBlock::FreeList::FreeList(FreeCell* head, size_t bytes)
+ : head(head)
+ , bytes(bytes)
+ {
+ }
+
inline size_t MarkedBlock::firstAtom()
{
return WTF::roundUpToMultipleOf<atomSize>(sizeof(MarkedBlock)) / atomSize;
diff --git a/Source/JavaScriptCore/heap/MarkedSpace.cpp b/Source/JavaScriptCore/heap/MarkedSpace.cpp
index a7d9da2..565fa86 100644
--- a/Source/JavaScriptCore/heap/MarkedSpace.cpp
+++ b/Source/JavaScriptCore/heap/MarkedSpace.cpp
@@ -31,8 +31,7 @@
class Structure;
MarkedSpace::MarkedSpace(Heap* heap)
- : m_waterMark(0)
- , m_heap(heap)
+ : m_heap(heap)
{
for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep) {
allocatorFor(cellSize).init(heap, this, cellSize, false);
@@ -47,8 +46,6 @@
void MarkedSpace::resetAllocators()
{
- m_waterMark = 0;
-
for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep) {
allocatorFor(cellSize).reset();
destructorAllocatorFor(cellSize).reset();
diff --git a/Source/JavaScriptCore/heap/MarkedSpace.h b/Source/JavaScriptCore/heap/MarkedSpace.h
index 2ac053e..a52d25f 100644
--- a/Source/JavaScriptCore/heap/MarkedSpace.h
+++ b/Source/JavaScriptCore/heap/MarkedSpace.h
@@ -65,9 +65,6 @@
void canonicalizeCellLivenessData();
- size_t waterMark();
- void addToWaterMark(size_t);
-
typedef HashSet<MarkedBlock*>::iterator BlockIterator;
template<typename Functor> typename Functor::ReturnType forEachCell(Functor&);
@@ -102,21 +99,10 @@
Subspace m_destructorSpace;
Subspace m_normalSpace;
- size_t m_waterMark;
Heap* m_heap;
MarkedBlockSet m_blocks;
};
-inline size_t MarkedSpace::waterMark()
-{
- return m_waterMark;
-}
-
-inline void MarkedSpace::addToWaterMark(size_t size)
-{
- m_waterMark += size;
-}
-
template<typename Functor> inline typename Functor::ReturnType MarkedSpace::forEachCell(Functor& functor)
{
canonicalizeCellLivenessData();
@@ -197,11 +183,6 @@
m_blocks.add(block);
}
-inline void MarkedSpace::didConsumeFreeList(MarkedBlock* block)
-{
- m_waterMark += block->capacity();
-}
-
} // namespace JSC
#endif // MarkedSpace_h
diff --git a/Source/JavaScriptCore/heap/WeakSet.cpp b/Source/JavaScriptCore/heap/WeakSet.cpp
index 423cc34..2878512 100644
--- a/Source/JavaScriptCore/heap/WeakSet.cpp
+++ b/Source/JavaScriptCore/heap/WeakSet.cpp
@@ -84,11 +84,15 @@
if (WeakBlock::FreeCell* allocator = tryFindAllocator())
return allocator;
- m_heap->addToWaterMark(WeakBlock::blockSize);
+ // FIXME: This reporting of the amount allocated isn't quite accurate and
+ // probably should be reworked eventually.
+ m_heap->didAllocate(WeakBlock::blockSize);
+ if (m_heap->shouldCollect()) {
+ m_heap->collect(Heap::DoNotSweep);
- // addToWaterMark() may cause a GC, so try again.
- if (WeakBlock::FreeCell* allocator = tryFindAllocator())
- return allocator;
+ if (WeakBlock::FreeCell* allocator = tryFindAllocator())
+ return allocator;
+ }
return addAllocator();
}
diff --git a/Source/JavaScriptCore/jit/JITInlineMethods.h b/Source/JavaScriptCore/jit/JITInlineMethods.h
index f2d0cef..5ff467c 100644
--- a/Source/JavaScriptCore/jit/JITInlineMethods.h
+++ b/Source/JavaScriptCore/jit/JITInlineMethods.h
@@ -414,12 +414,12 @@
allocator = &m_globalData->heap.allocatorForObjectWithDestructor(sizeof(ClassType));
else
allocator = &m_globalData->heap.allocatorForObjectWithoutDestructor(sizeof(ClassType));
- loadPtr(&allocator->m_firstFreeCell, result);
+ loadPtr(&allocator->m_freeList.head, result);
addSlowCase(branchTestPtr(Zero, result));
// remove the object from the free list
loadPtr(Address(result), storagePtr);
- storePtr(storagePtr, &allocator->m_firstFreeCell);
+ storePtr(storagePtr, &allocator->m_freeList.head);
// initialize the object's structure
storePtr(structure, Address(result, JSCell::structureOffset()));
diff --git a/Source/JavaScriptCore/llint/LowLevelInterpreter.asm b/Source/JavaScriptCore/llint/LowLevelInterpreter.asm
index 956e14a..bbfa859 100644
--- a/Source/JavaScriptCore/llint/LowLevelInterpreter.asm
+++ b/Source/JavaScriptCore/llint/LowLevelInterpreter.asm
@@ -291,17 +291,21 @@
MarkedSpace::Subspace::preciseAllocators +
sizeClassIndex * sizeof MarkedAllocator
+ const offsetOfFirstFreeCell =
+ MarkedAllocator::m_freeList +
+ MarkedBlock::FreeList::head
+
# FIXME: we can get the global data in one load from the stack.
loadp CodeBlock[cfr], scratch1
loadp CodeBlock::m_globalData[scratch1], scratch1
- # Get the object from the free list.
- loadp offsetOfMySizeClass + MarkedAllocator::m_firstFreeCell[scratch1], result
+ # Get the object from the free list.
+ loadp offsetOfMySizeClass + offsetOfFirstFreeCell[scratch1], result
btpz result, slowCase
# Remove the object from the free list.
loadp [result], scratch2
- storep scratch2, offsetOfMySizeClass + MarkedAllocator::m_firstFreeCell[scratch1]
+ storep scratch2, offsetOfMySizeClass + offsetOfFirstFreeCell[scratch1]
# Initialize the object.
loadp classInfoOffset[scratch1], scratch2
diff --git a/Source/JavaScriptCore/runtime/GCActivityCallback.cpp b/Source/JavaScriptCore/runtime/GCActivityCallback.cpp
index a40a7af..0946c48 100644
--- a/Source/JavaScriptCore/runtime/GCActivityCallback.cpp
+++ b/Source/JavaScriptCore/runtime/GCActivityCallback.cpp
@@ -42,7 +42,7 @@
{
}
-void DefaultGCActivityCallback::willAllocate()
+void DefaultGCActivityCallback::didAllocate(size_t)
{
}
diff --git a/Source/JavaScriptCore/runtime/GCActivityCallback.h b/Source/JavaScriptCore/runtime/GCActivityCallback.h
index cf06304..11486d8 100644
--- a/Source/JavaScriptCore/runtime/GCActivityCallback.h
+++ b/Source/JavaScriptCore/runtime/GCActivityCallback.h
@@ -43,7 +43,7 @@
class GCActivityCallback {
public:
virtual ~GCActivityCallback() { }
- virtual void willAllocate() { }
+ virtual void didAllocate(size_t) { }
virtual void didCollect() { }
virtual void didAbandonObjectGraph() { }
virtual void synchronize() { }
@@ -61,7 +61,7 @@
DefaultGCActivityCallback(Heap*);
virtual ~DefaultGCActivityCallback();
- virtual void willAllocate();
+ virtual void didAllocate(size_t);
virtual void didCollect();
virtual void didAbandonObjectGraph();
virtual void synchronize();
diff --git a/Source/JavaScriptCore/runtime/GCActivityCallbackCF.cpp b/Source/JavaScriptCore/runtime/GCActivityCallbackCF.cpp
index 3fdb69b..2973a0d 100644
--- a/Source/JavaScriptCore/runtime/GCActivityCallbackCF.cpp
+++ b/Source/JavaScriptCore/runtime/GCActivityCallbackCF.cpp
@@ -57,6 +57,7 @@
const double gcTimerIntervalMultiplier = 1.0 / gcCPUBudget;
const CFTimeInterval decade = 60 * 60 * 24 * 365 * 10;
const CFTimeInterval hour = 60 * 60;
+const size_t minBytesBeforeCollect = 128 * KB;
void DefaultGCActivityCallbackPlatformData::timerDidFire(CFRunLoopTimerRef, void *info)
{
@@ -113,8 +114,10 @@
CFRunLoopTimerSetNextFireDate(d->timer.get(), CFAbsoluteTimeGetCurrent() + decade);
}
-void DefaultGCActivityCallback::willAllocate()
+void DefaultGCActivityCallback::didAllocate(size_t bytes)
{
+ if (bytes < minBytesBeforeCollect)
+ return;
scheduleTimer(d.get());
}
