Get rid of TLCs
https://bugs.webkit.org/show_bug.cgi?id=185846
Rubber stamped by Geoffrey Garen.
Source/JavaScriptCore:
This removes support for thread-local caches from the GC in order to speed up allocation a
bit.
We added TLCs as part of Spectre mitigations, which we have since removed.
We will want some kind of TLCs eventually, since they allow us to:
- have a global GC, which may be a perf optimization at some point.
- allocate objects from JIT threads, which we've been wanting to do for a while.
This change keeps the most interesting aspect of TLCs, which is the
LocalAllocator/BlockDirectory separation. This means that it ought to be easy to implement
TLCs again in the future if we wanted this feature.
This change removes the part of TLCs that causes a perf regression, namely that Allocator is
an offset that requires a bounds check and lookup that makes the rest of the allocation fast
path dependent on the load of the TLC. Now, Allocator is really just a LocalAllocator*, so
you can directly use it to allocate. This removes two loads and a check from the allocation
fast path. In hindsight, I probably could have made that whole thing more efficient, had I
allowed us to have a statically known set of LocalAllocators. This would have removed the
bounds check (one load and one branch) and it would have made it possible to CSE the load of
the TLC data structure, since that would no longer resize. But that's a harder change that
this patch, and we don't need it right now.
While reviewing the allocation hot paths, I found that CreateThis had an unnecessary branch
to check if the allocator is null. I removed that check. AssemblyHelpers::emitAllocate() does
that check already. Previously, the TLC bounds check doubled as this check.
This is a 1% speed-up on Octane and a 2.3% speed-up on TailBench. However, the Octane
speed-up on my machine includes an 8% regexp speed-up. I've found that sometimes regexp
speeds up or slows down by 8% depending on which path I build JSC from. Without that 8%, this
is still an Octane speed-up due to 2-4% speed-ups in earley, boyer, raytrace, and splay.
* JavaScriptCore.xcodeproj/project.pbxproj:
* Sources.txt:
* bytecode/ObjectAllocationProfileInlines.h:
(JSC::ObjectAllocationProfile::initializeProfile):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileCreateThis):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
* heap/Allocator.cpp:
(JSC::Allocator::cellSize const):
* heap/Allocator.h:
(JSC::Allocator::Allocator):
(JSC::Allocator::localAllocator const):
(JSC::Allocator::operator== const):
(JSC::Allocator::offset const): Deleted.
* heap/AllocatorInlines.h:
(JSC::Allocator::allocate const):
(JSC::Allocator::tryAllocate const): Deleted.
* heap/BlockDirectory.cpp:
(JSC::BlockDirectory::BlockDirectory):
(JSC::BlockDirectory::~BlockDirectory):
* heap/BlockDirectory.h:
(JSC::BlockDirectory::allocator const): Deleted.
* heap/CompleteSubspace.cpp:
(JSC::CompleteSubspace::allocateNonVirtual):
(JSC::CompleteSubspace::allocatorForSlow):
(JSC::CompleteSubspace::tryAllocateSlow):
* heap/CompleteSubspace.h:
* heap/Heap.cpp:
(JSC::Heap::Heap):
* heap/Heap.h:
(JSC::Heap::threadLocalCacheLayout): Deleted.
* heap/IsoSubspace.cpp:
(JSC::IsoSubspace::IsoSubspace):
(JSC::IsoSubspace::allocateNonVirtual):
* heap/IsoSubspace.h:
(JSC::IsoSubspace::allocatorForNonVirtual):
* heap/LocalAllocator.cpp:
(JSC::LocalAllocator::LocalAllocator):
(JSC::LocalAllocator::~LocalAllocator):
* heap/LocalAllocator.h:
(JSC::LocalAllocator::cellSize const):
(JSC::LocalAllocator::tlc const): Deleted.
* heap/ThreadLocalCache.cpp: Removed.
* heap/ThreadLocalCache.h: Removed.
* heap/ThreadLocalCacheInlines.h: Removed.
* heap/ThreadLocalCacheLayout.cpp: Removed.
* heap/ThreadLocalCacheLayout.h: Removed.
* jit/AssemblyHelpers.cpp:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_create_this):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
* runtime/VM.h:
* runtime/VMEntryScope.cpp:
(JSC::VMEntryScope::~VMEntryScope):
* runtime/VMEntryScope.h:
Source/WTF:
* wtf/Platform.h:
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@232074 268f45cc-cd09-0410-ab3c-d52691b4dbfc
diff --git a/Source/JavaScriptCore/CMakeLists.txt b/Source/JavaScriptCore/CMakeLists.txt
index 5e56d50..8149188 100644
--- a/Source/JavaScriptCore/CMakeLists.txt
+++ b/Source/JavaScriptCore/CMakeLists.txt
@@ -556,7 +556,6 @@
heap/Subspace.h
heap/SubspaceInlines.h
heap/Synchronousness.h
- heap/ThreadLocalCache.h
heap/TinyBloomFilter.h
heap/UnconditionalFinalizer.h
heap/VisitRaceKey.h
diff --git a/Source/JavaScriptCore/ChangeLog b/Source/JavaScriptCore/ChangeLog
index 20484aa..bb6a5a8 100644
--- a/Source/JavaScriptCore/ChangeLog
+++ b/Source/JavaScriptCore/ChangeLog
@@ -1,3 +1,112 @@
+2018-05-21 Filip Pizlo <fpizlo@apple.com>
+
+ Get rid of TLCs
+ https://bugs.webkit.org/show_bug.cgi?id=185846
+
+ Rubber stamped by Geoffrey Garen.
+
+ This removes support for thread-local caches from the GC in order to speed up allocation a
+ bit.
+
+ We added TLCs as part of Spectre mitigations, which we have since removed.
+
+ We will want some kind of TLCs eventually, since they allow us to:
+
+ - have a global GC, which may be a perf optimization at some point.
+ - allocate objects from JIT threads, which we've been wanting to do for a while.
+
+ This change keeps the most interesting aspect of TLCs, which is the
+ LocalAllocator/BlockDirectory separation. This means that it ought to be easy to implement
+ TLCs again in the future if we wanted this feature.
+
+ This change removes the part of TLCs that causes a perf regression, namely that Allocator is
+ an offset that requires a bounds check and lookup that makes the rest of the allocation fast
+ path dependent on the load of the TLC. Now, Allocator is really just a LocalAllocator*, so
+ you can directly use it to allocate. This removes two loads and a check from the allocation
+ fast path. In hindsight, I probably could have made that whole thing more efficient, had I
+ allowed us to have a statically known set of LocalAllocators. This would have removed the
+ bounds check (one load and one branch) and it would have made it possible to CSE the load of
+ the TLC data structure, since that would no longer resize. But that's a harder change that
+ this patch, and we don't need it right now.
+
+ While reviewing the allocation hot paths, I found that CreateThis had an unnecessary branch
+ to check if the allocator is null. I removed that check. AssemblyHelpers::emitAllocate() does
+ that check already. Previously, the TLC bounds check doubled as this check.
+
+ This is a 1% speed-up on Octane and a 2.3% speed-up on TailBench. However, the Octane
+ speed-up on my machine includes an 8% regexp speed-up. I've found that sometimes regexp
+ speeds up or slows down by 8% depending on which path I build JSC from. Without that 8%, this
+ is still an Octane speed-up due to 2-4% speed-ups in earley, boyer, raytrace, and splay.
+
+ * JavaScriptCore.xcodeproj/project.pbxproj:
+ * Sources.txt:
+ * bytecode/ObjectAllocationProfileInlines.h:
+ (JSC::ObjectAllocationProfile::initializeProfile):
+ * dfg/DFGSpeculativeJIT.cpp:
+ (JSC::DFG::SpeculativeJIT::compileCreateThis):
+ * ftl/FTLLowerDFGToB3.cpp:
+ (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
+ (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
+ (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
+ (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
+ (JSC::FTL::DFG::LowerDFGToB3::allocateObject):
+ (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
+ * heap/Allocator.cpp:
+ (JSC::Allocator::cellSize const):
+ * heap/Allocator.h:
+ (JSC::Allocator::Allocator):
+ (JSC::Allocator::localAllocator const):
+ (JSC::Allocator::operator== const):
+ (JSC::Allocator::offset const): Deleted.
+ * heap/AllocatorInlines.h:
+ (JSC::Allocator::allocate const):
+ (JSC::Allocator::tryAllocate const): Deleted.
+ * heap/BlockDirectory.cpp:
+ (JSC::BlockDirectory::BlockDirectory):
+ (JSC::BlockDirectory::~BlockDirectory):
+ * heap/BlockDirectory.h:
+ (JSC::BlockDirectory::allocator const): Deleted.
+ * heap/CompleteSubspace.cpp:
+ (JSC::CompleteSubspace::allocateNonVirtual):
+ (JSC::CompleteSubspace::allocatorForSlow):
+ (JSC::CompleteSubspace::tryAllocateSlow):
+ * heap/CompleteSubspace.h:
+ * heap/Heap.cpp:
+ (JSC::Heap::Heap):
+ * heap/Heap.h:
+ (JSC::Heap::threadLocalCacheLayout): Deleted.
+ * heap/IsoSubspace.cpp:
+ (JSC::IsoSubspace::IsoSubspace):
+ (JSC::IsoSubspace::allocateNonVirtual):
+ * heap/IsoSubspace.h:
+ (JSC::IsoSubspace::allocatorForNonVirtual):
+ * heap/LocalAllocator.cpp:
+ (JSC::LocalAllocator::LocalAllocator):
+ (JSC::LocalAllocator::~LocalAllocator):
+ * heap/LocalAllocator.h:
+ (JSC::LocalAllocator::cellSize const):
+ (JSC::LocalAllocator::tlc const): Deleted.
+ * heap/ThreadLocalCache.cpp: Removed.
+ * heap/ThreadLocalCache.h: Removed.
+ * heap/ThreadLocalCacheInlines.h: Removed.
+ * heap/ThreadLocalCacheLayout.cpp: Removed.
+ * heap/ThreadLocalCacheLayout.h: Removed.
+ * jit/AssemblyHelpers.cpp:
+ (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
+ (JSC::AssemblyHelpers::emitAllocate):
+ (JSC::AssemblyHelpers::emitAllocateVariableSized):
+ * jit/JITOpcodes.cpp:
+ (JSC::JIT::emit_op_create_this):
+ * runtime/JSLock.cpp:
+ (JSC::JSLock::didAcquireLock):
+ * runtime/VM.cpp:
+ (JSC::VM::VM):
+ (JSC::VM::~VM):
+ * runtime/VM.h:
+ * runtime/VMEntryScope.cpp:
+ (JSC::VMEntryScope::~VMEntryScope):
+ * runtime/VMEntryScope.h:
+
2018-05-22 Keith Miller <keith_miller@apple.com>
We should have a CoW storage for NewArrayBuffer arrays.
diff --git a/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj b/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
index 0e4cb35..74a6e6f 100644
--- a/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
+++ b/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
@@ -395,12 +395,10 @@
0F725CAA1C503DED00AD943A /* B3PureCSE.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F725CA61C503DED00AD943A /* B3PureCSE.h */; };
0F725CB01C506D3B00AD943A /* B3FoldPathConstants.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F725CAE1C506D3B00AD943A /* B3FoldPathConstants.h */; };
0F74B93B1F89614800B935D3 /* PrototypeKey.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F74B93A1F89614500B935D3 /* PrototypeKey.h */; settings = {ATTRIBUTES = (Private, ); }; };
- 0F75A05E200D25F60038E2CF /* ThreadLocalCache.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A055200D25EF0038E2CF /* ThreadLocalCache.h */; settings = {ATTRIBUTES = (Private, ); }; };
0F75A060200D260B0038E2CF /* LocalAllocatorInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A05A200D25F00038E2CF /* LocalAllocatorInlines.h */; };
0F75A061200D26180038E2CF /* LocalAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A057200D25F00038E2CF /* LocalAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
0F75A062200D261D0038E2CF /* AllocatorInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A05D200D25F10038E2CF /* AllocatorInlines.h */; };
0F75A063200D261F0038E2CF /* Allocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A054200D25EF0038E2CF /* Allocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
- 0F75A064200D26280038E2CF /* ThreadLocalCacheLayout.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A05C200D25F10038E2CF /* ThreadLocalCacheLayout.h */; settings = {ATTRIBUTES = (Private, ); }; };
0F75A0662013E4F10038E2CF /* JITAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A0652013E4EF0038E2CF /* JITAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
0F766D2C15A8CC3A008F363E /* JITStubRoutineSet.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F766D2A15A8CC34008F363E /* JITStubRoutineSet.h */; settings = {ATTRIBUTES = (Private, ); }; };
0F766D3015A8DCE2008F363E /* GCAwareJITStubRoutine.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F766D2E15A8DCDD008F363E /* GCAwareJITStubRoutine.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -2450,14 +2448,9 @@
0F725CAE1C506D3B00AD943A /* B3FoldPathConstants.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3FoldPathConstants.h; path = b3/B3FoldPathConstants.h; sourceTree = "<group>"; };
0F74B93A1F89614500B935D3 /* PrototypeKey.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = PrototypeKey.h; sourceTree = "<group>"; };
0F75A054200D25EF0038E2CF /* Allocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Allocator.h; sourceTree = "<group>"; };
- 0F75A055200D25EF0038E2CF /* ThreadLocalCache.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ThreadLocalCache.h; sourceTree = "<group>"; };
- 0F75A056200D25EF0038E2CF /* ThreadLocalCacheInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ThreadLocalCacheInlines.h; sourceTree = "<group>"; };
0F75A057200D25F00038E2CF /* LocalAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LocalAllocator.h; sourceTree = "<group>"; };
- 0F75A058200D25F00038E2CF /* ThreadLocalCache.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ThreadLocalCache.cpp; sourceTree = "<group>"; };
0F75A059200D25F00038E2CF /* LocalAllocator.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LocalAllocator.cpp; sourceTree = "<group>"; };
0F75A05A200D25F00038E2CF /* LocalAllocatorInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LocalAllocatorInlines.h; sourceTree = "<group>"; };
- 0F75A05B200D25F10038E2CF /* ThreadLocalCacheLayout.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ThreadLocalCacheLayout.cpp; sourceTree = "<group>"; };
- 0F75A05C200D25F10038E2CF /* ThreadLocalCacheLayout.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ThreadLocalCacheLayout.h; sourceTree = "<group>"; };
0F75A05D200D25F10038E2CF /* AllocatorInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AllocatorInlines.h; sourceTree = "<group>"; };
0F75A0652013E4EF0038E2CF /* JITAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITAllocator.h; sourceTree = "<group>"; };
0F766D1C15A5028D008F363E /* JITStubRoutine.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITStubRoutine.h; sourceTree = "<group>"; };
@@ -5779,11 +5772,6 @@
0FD79A2C1EBBBDB200DA88D3 /* Synchronousness.h */,
0F1FB38A1E173A6200A9BE50 /* SynchronousStopTheWorldMutatorScheduler.cpp */,
0F1FB38B1E173A6200A9BE50 /* SynchronousStopTheWorldMutatorScheduler.h */,
- 0F75A058200D25F00038E2CF /* ThreadLocalCache.cpp */,
- 0F75A055200D25EF0038E2CF /* ThreadLocalCache.h */,
- 0F75A056200D25EF0038E2CF /* ThreadLocalCacheInlines.h */,
- 0F75A05B200D25F10038E2CF /* ThreadLocalCacheLayout.cpp */,
- 0F75A05C200D25F10038E2CF /* ThreadLocalCacheLayout.h */,
141448CC13A1783700F5BA1A /* TinyBloomFilter.h */,
0F5F08CE146C762F000472A9 /* UnconditionalFinalizer.h */,
0F4D8C721FC7A973001D32AC /* VisitCounter.h */,
@@ -9471,8 +9459,6 @@
70ECA6091AFDBEA200449739 /* TemplateObjectDescriptor.h in Headers */,
0F24E54F17EE274900ABB217 /* TempRegisterSet.h in Headers */,
0FA2C17C17D7CF84009D015F /* TestRunnerUtils.h in Headers */,
- 0F75A05E200D25F60038E2CF /* ThreadLocalCache.h in Headers */,
- 0F75A064200D26280038E2CF /* ThreadLocalCacheLayout.h in Headers */,
FE3422121D6B81C30032BE88 /* ThrowScope.h in Headers */,
0F572D4F16879FDD00E57FBD /* ThunkGenerator.h in Headers */,
A7386556118697B400540279 /* ThunkGenerators.h in Headers */,
diff --git a/Source/JavaScriptCore/Sources.txt b/Source/JavaScriptCore/Sources.txt
index 0b15189..c011ee6 100644
--- a/Source/JavaScriptCore/Sources.txt
+++ b/Source/JavaScriptCore/Sources.txt
@@ -524,8 +524,6 @@
heap/Subspace.cpp
heap/SynchronousStopTheWorldMutatorScheduler.cpp
heap/Synchronousness.cpp
-heap/ThreadLocalCache.cpp
-heap/ThreadLocalCacheLayout.cpp
heap/VisitRaceKey.cpp
heap/Weak.cpp
heap/WeakBlock.cpp
diff --git a/Source/JavaScriptCore/bytecode/ObjectAllocationProfileInlines.h b/Source/JavaScriptCore/bytecode/ObjectAllocationProfileInlines.h
index 553ac51..106dab1 100644
--- a/Source/JavaScriptCore/bytecode/ObjectAllocationProfileInlines.h
+++ b/Source/JavaScriptCore/bytecode/ObjectAllocationProfileInlines.h
@@ -105,7 +105,7 @@
// Take advantage of extra inline capacity available in the size class.
if (allocator) {
- size_t slop = (allocator.cellSize(vm.heap) - allocationSize) / sizeof(WriteBarrier<Unknown>);
+ size_t slop = (allocator.cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
inlineCapacity += slop;
if (inlineCapacity > JSFinalObject::maxInlineCapacity())
inlineCapacity = JSFinalObject::maxInlineCapacity();
diff --git a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
index d18b102..3d5ebfe 100644
--- a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
+++ b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
@@ -12196,11 +12196,9 @@
m_jit.loadPtr(JITCompiler::Address(calleeGPR, JSFunction::offsetOfRareData()), rareDataGPR);
slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, rareDataGPR));
m_jit.xorPtr(JITCompiler::TrustedImmPtr(JSFunctionPoison::key()), rareDataGPR);
- m_jit.load32(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
+ m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
- slowPath.append(m_jit.branch32(MacroAssembler::Equal, allocatorGPR, TrustedImm32(Allocator().offset())));
-
auto butterfly = TrustedImmPtr(nullptr);
emitAllocateJSObject(resultGPR, JITAllocator::variable(), allocatorGPR, structureGPR, butterfly, scratchGPR, slowPath);
diff --git a/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp b/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
index d98a8d8..563de2b 100644
--- a/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
+++ b/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
@@ -6165,7 +6165,7 @@
Allocator allocator = subspaceFor<JSRopeString>(vm())->allocatorForNonVirtual(sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
LValue result = allocateCell(
- m_out.constInt32(allocator.offset()), vm().stringStructure.get(), slowPath);
+ m_out.constIntPtr(allocator.localAllocator()), vm().stringStructure.get(), slowPath);
m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSString_value);
for (unsigned i = 0; i < numKids; ++i)
@@ -10467,7 +10467,8 @@
m_out.store32(vectorLength, fastButterflyValue, m_heaps.Butterfly_vectorLength);
LValue fastObjectValue = allocateObject(
- m_out.constInt32(cellAllocator.offset()), structure, fastButterflyValue, slowPath);
+ m_out.constIntPtr(cellAllocator.localAllocator()), structure, fastButterflyValue,
+ slowPath);
ValueFromBlock fastObject = m_out.anchor(fastObjectValue);
ValueFromBlock fastButterfly = m_out.anchor(fastButterflyValue);
@@ -11469,7 +11470,8 @@
size_t sizeInBytes = sizeInValues * sizeof(JSValue);
Allocator allocator = vm().jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(sizeInBytes, AllocatorForMode::AllocatorIfExists);
- LValue startOfStorage = allocateHeapCell(m_out.constInt32(allocator.offset()), slowPath);
+ LValue startOfStorage = allocateHeapCell(
+ m_out.constIntPtr(allocator.localAllocator()), slowPath);
ValueFromBlock fastButterfly = m_out.anchor(
m_out.add(m_out.constIntPtr(sizeInBytes + sizeof(IndexingHeader)), startOfStorage));
m_out.jump(continuation);
@@ -12513,7 +12515,7 @@
{
JITAllocator actualAllocator;
if (allocator->hasInt32())
- actualAllocator = JITAllocator::constant(Allocator(allocator->asInt32()));
+ actualAllocator = JITAllocator::constant(Allocator(bitwise_cast<LocalAllocator*>(allocator->asIntPtr())));
else
actualAllocator = JITAllocator::variable();
@@ -12531,7 +12533,7 @@
LBasicBlock haveAllocator = m_out.newBlock();
LBasicBlock lastNext = m_out.insertNewBlocksBefore(haveAllocator);
m_out.branch(
- m_out.notEqual(allocator, m_out.constInt32(Allocator().offset())),
+ m_out.notEqual(allocator, m_out.intPtrZero),
usually(haveAllocator), rarely(slowPath));
m_out.appendTo(haveAllocator, lastNext);
}
@@ -12661,7 +12663,8 @@
size_t size, StructureType structure, LValue butterfly, LBasicBlock slowPath)
{
Allocator allocator = subspaceFor<ClassType>(vm())->allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
- return allocateObject(m_out.constInt32(allocator.offset()), structure, butterfly, slowPath);
+ return allocateObject(
+ m_out.constIntPtr(allocator.localAllocator()), structure, butterfly, slowPath);
}
template<typename ClassType, typename StructureType>
@@ -12686,10 +12689,10 @@
LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
m_out.jump(slowPath);
m_out.appendTo(continuation, lastNext);
- return m_out.int32Zero;
+ return m_out.intPtrZero;
}
- return m_out.constInt32(actualAllocator.offset());
+ return m_out.constIntPtr(actualAllocator.localAllocator());
}
unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
@@ -12708,7 +12711,7 @@
m_out.appendTo(continuation, lastNext);
- return m_out.load32(
+ return m_out.loadPtr(
m_out.baseIndex(
m_heaps.CompleteSubspace_allocatorForSizeStep,
subspace, m_out.sub(sizeClassIndex, m_out.intPtrOne)));
@@ -12750,7 +12753,7 @@
LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
ValueFromBlock fastResult = m_out.anchor(allocateObject(
- m_out.constInt32(allocator.offset()), structure, m_out.intPtrZero, slowPath));
+ m_out.constIntPtr(allocator.localAllocator()), structure, m_out.intPtrZero, slowPath));
m_out.jump(continuation);
diff --git a/Source/JavaScriptCore/heap/Allocator.cpp b/Source/JavaScriptCore/heap/Allocator.cpp
index b1281e4..6e5964e2 100644
--- a/Source/JavaScriptCore/heap/Allocator.cpp
+++ b/Source/JavaScriptCore/heap/Allocator.cpp
@@ -27,13 +27,13 @@
#include "Allocator.h"
#include "Heap.h"
-#include "ThreadLocalCacheLayout.h"
+#include "LocalAllocator.h"
namespace JSC {
-unsigned Allocator::cellSize(Heap& heap) const
+unsigned Allocator::cellSize() const
{
- return heap.threadLocalCacheLayout().directory(m_offset)->cellSize();
+ return m_localAllocator->cellSize();
}
} // namespace JSC
diff --git a/Source/JavaScriptCore/heap/Allocator.h b/Source/JavaScriptCore/heap/Allocator.h
index 7c3da48..229158f 100644
--- a/Source/JavaScriptCore/heap/Allocator.h
+++ b/Source/JavaScriptCore/heap/Allocator.h
@@ -31,35 +31,32 @@
namespace JSC {
class GCDeferralContext;
-class Heap;
-class VM;
+class LocalAllocator;
+
+// This abstracts how we refer to LocalAllocator so that we could eventually support thread-local
+// caches.
class Allocator {
public:
Allocator() { }
- explicit Allocator(unsigned offset)
- : m_offset(offset)
+ explicit Allocator(LocalAllocator* localAllocator)
+ : m_localAllocator(localAllocator)
{
}
- void* allocate(VM&, GCDeferralContext*, AllocationFailureMode) const;
+ void* allocate(GCDeferralContext*, AllocationFailureMode) const;
- // This version calls FailureFunc if we have a null allocator or if the TLC hasn't been resized
- // to include this allocator.
- template<typename FailureFunc>
- void* tryAllocate(VM&, GCDeferralContext*, AllocationFailureMode, const FailureFunc&) const;
+ unsigned cellSize() const;
- unsigned cellSize(Heap&) const;
+ LocalAllocator* localAllocator() const { return m_localAllocator; }
- unsigned offset() const { return m_offset; }
-
- bool operator==(const Allocator& other) const { return m_offset == other.offset(); }
+ bool operator==(const Allocator& other) const { return m_localAllocator == other.localAllocator(); }
bool operator!=(const Allocator& other) const { return !(*this == other); }
explicit operator bool() const { return *this != Allocator(); }
private:
- unsigned m_offset { UINT_MAX };
+ LocalAllocator* m_localAllocator { nullptr };
};
} // namespace JSC
diff --git a/Source/JavaScriptCore/heap/AllocatorInlines.h b/Source/JavaScriptCore/heap/AllocatorInlines.h
index 5f0c73e..3005e23 100644
--- a/Source/JavaScriptCore/heap/AllocatorInlines.h
+++ b/Source/JavaScriptCore/heap/AllocatorInlines.h
@@ -26,28 +26,13 @@
#pragma once
#include "Allocator.h"
-#include "ThreadLocalCache.h"
+#include "LocalAllocator.h"
namespace JSC {
-inline void* Allocator::allocate(VM& vm, GCDeferralContext* context, AllocationFailureMode mode) const
+inline void* Allocator::allocate(GCDeferralContext* context, AllocationFailureMode mode) const
{
- return ThreadLocalCache::allocator(vm, m_offset).allocate(context, mode);
-}
-
-template<typename FailureFunc>
-void* Allocator::tryAllocate(VM& vm, GCDeferralContext* context, AllocationFailureMode mode, const FailureFunc& failureFunc) const
-{
- void* result;
- ThreadLocalCache::tryGetAllocator(
- vm, m_offset,
- [&] (LocalAllocator& allocator) {
- result = allocator.allocate(context, mode);
- },
- [&] () {
- result = failureFunc();
- });
- return result;
+ return m_localAllocator->allocate(context, mode);
}
} // namespace JSC
diff --git a/Source/JavaScriptCore/heap/BlockDirectory.cpp b/Source/JavaScriptCore/heap/BlockDirectory.cpp
index 63e1344..f5f2b3e 100644
--- a/Source/JavaScriptCore/heap/BlockDirectory.cpp
+++ b/Source/JavaScriptCore/heap/BlockDirectory.cpp
@@ -33,7 +33,6 @@
#include "JSCInlines.h"
#include "MarkedBlockInlines.h"
#include "SuperSampler.h"
-#include "ThreadLocalCacheInlines.h"
#include "VM.h"
namespace JSC {
@@ -42,11 +41,13 @@
: m_cellSize(static_cast<unsigned>(cellSize))
, m_heap(heap)
{
- heap->threadLocalCacheLayout().allocateOffset(this);
}
BlockDirectory::~BlockDirectory()
{
+ auto locker = holdLock(m_localAllocatorsLock);
+ while (!m_localAllocators.isEmpty())
+ m_localAllocators.begin()->remove();
}
void BlockDirectory::setSubspace(Subspace* subspace)
diff --git a/Source/JavaScriptCore/heap/BlockDirectory.h b/Source/JavaScriptCore/heap/BlockDirectory.h
index 5960f53..a12b433 100644
--- a/Source/JavaScriptCore/heap/BlockDirectory.h
+++ b/Source/JavaScriptCore/heap/BlockDirectory.h
@@ -26,7 +26,6 @@
#pragma once
#include "AllocationFailureMode.h"
-#include "Allocator.h"
#include "CellAttributes.h"
#include "FreeList.h"
#include "LocalAllocator.h"
@@ -44,8 +43,6 @@
class IsoCellSet;
class MarkedSpace;
class LLIntOffsetsExtractor;
-class ThreadLocalCache;
-class ThreadLocalCacheLayout;
#define FOR_EACH_BLOCK_DIRECTORY_BIT(macro) \
macro(live, Live) /* The set of block indices that have actual blocks. */\
@@ -158,8 +155,6 @@
Subspace* subspace() const { return m_subspace; }
MarkedSpace& markedSpace() const;
- Allocator allocator() const { return Allocator(m_tlcOffset); }
-
void dump(PrintStream&) const;
void dumpBits(PrintStream& = WTF::dataFile());
@@ -168,7 +163,6 @@
friend class LocalAllocator;
friend class LocalSideAllocator;
friend class MarkedBlock;
- friend class ThreadLocalCacheLayout;
MarkedBlock::Handle* findBlockForAllocation(LocalAllocator&);
@@ -201,7 +195,6 @@
BlockDirectory* m_nextDirectoryInAlignedMemoryAllocator { nullptr };
Lock m_localAllocatorsLock;
- size_t m_tlcOffset;
SentinelLinkedList<LocalAllocator, BasicRawSentinelNode<LocalAllocator>> m_localAllocators;
};
diff --git a/Source/JavaScriptCore/heap/CompleteSubspace.cpp b/Source/JavaScriptCore/heap/CompleteSubspace.cpp
index bb04c54..de40a3d 100644
--- a/Source/JavaScriptCore/heap/CompleteSubspace.cpp
+++ b/Source/JavaScriptCore/heap/CompleteSubspace.cpp
@@ -34,7 +34,6 @@
#include "MarkedBlockInlines.h"
#include "PreventCollectionScope.h"
#include "SubspaceInlines.h"
-#include "ThreadLocalCacheInlines.h"
namespace JSC {
@@ -60,12 +59,9 @@
void* CompleteSubspace::allocateNonVirtual(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
{
- Allocator allocator = allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
- return allocator.tryAllocate(
- vm, deferralContext, failureMode,
- [&] () {
- return allocateSlow(vm, size, deferralContext, failureMode);
- });
+ if (Allocator allocator = allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists))
+ return allocator.allocate(deferralContext, failureMode);
+ return allocateSlow(vm, size, deferralContext, failureMode);
}
Allocator CompleteSubspace::allocatorForSlow(size_t size)
@@ -88,28 +84,39 @@
return allocator;
if (false)
- dataLog("Creating marked allocator for ", m_name, ", ", m_attributes, ", ", sizeClass, ".\n");
+ dataLog("Creating BlockDirectory/LocalAllocator for ", m_name, ", ", m_attributes, ", ", sizeClass, ".\n");
+
std::unique_ptr<BlockDirectory> uniqueDirectory =
std::make_unique<BlockDirectory>(m_space.heap(), sizeClass);
BlockDirectory* directory = uniqueDirectory.get();
m_directories.append(WTFMove(uniqueDirectory));
+
directory->setSubspace(this);
m_space.addBlockDirectory(locker, directory);
+
+ std::unique_ptr<LocalAllocator> uniqueLocalAllocator =
+ std::make_unique<LocalAllocator>(directory);
+ LocalAllocator* localAllocator = uniqueLocalAllocator.get();
+ m_localAllocators.append(WTFMove(uniqueLocalAllocator));
+
+ Allocator allocator(localAllocator);
+
index = MarkedSpace::sizeClassToIndex(sizeClass);
for (;;) {
if (MarkedSpace::s_sizeClassForSizeStep[index] != sizeClass)
break;
- m_allocatorForSizeStep[index] = directory->allocator();
+ m_allocatorForSizeStep[index] = allocator;
if (!index--)
break;
}
+
directory->setNextDirectoryInSubspace(m_firstDirectory);
m_alignedMemoryAllocator->registerDirectory(directory);
WTF::storeStoreFence();
m_firstDirectory = directory;
- return directory->allocator();
+ return allocator;
}
void* CompleteSubspace::allocateSlow(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
@@ -125,7 +132,7 @@
sanitizeStackForVM(&vm);
if (Allocator allocator = allocatorFor(size, AllocatorForMode::EnsureAllocator))
- return allocator.allocate(vm, deferralContext, AllocationFailureMode::ReturnNull);
+ return allocator.allocate(deferralContext, AllocationFailureMode::ReturnNull);
if (size <= Options::largeAllocationCutoff()
&& size <= MarkedSpace::largeCutoff) {
diff --git a/Source/JavaScriptCore/heap/CompleteSubspace.h b/Source/JavaScriptCore/heap/CompleteSubspace.h
index e0c30d4..2010cd2 100644
--- a/Source/JavaScriptCore/heap/CompleteSubspace.h
+++ b/Source/JavaScriptCore/heap/CompleteSubspace.h
@@ -58,6 +58,7 @@
std::array<Allocator, MarkedSpace::numSizeClasses> m_allocatorForSizeStep;
Vector<std::unique_ptr<BlockDirectory>> m_directories;
+ Vector<std::unique_ptr<LocalAllocator>> m_localAllocators;
};
ALWAYS_INLINE Allocator CompleteSubspace::allocatorForNonVirtual(size_t size, AllocatorForMode mode)
diff --git a/Source/JavaScriptCore/heap/Heap.cpp b/Source/JavaScriptCore/heap/Heap.cpp
index 0787119..05bd9b5 100644
--- a/Source/JavaScriptCore/heap/Heap.cpp
+++ b/Source/JavaScriptCore/heap/Heap.cpp
@@ -68,7 +68,6 @@
#include "StopIfNecessaryTimer.h"
#include "SweepingScope.h"
#include "SynchronousStopTheWorldMutatorScheduler.h"
-#include "ThreadLocalCacheLayout.h"
#include "TypeProfiler.h"
#include "TypeProfilerLog.h"
#include "UnlinkedCodeBlock.h"
@@ -314,7 +313,6 @@
, m_helperClient(&heapHelperPool())
, m_threadLock(Box<Lock>::create())
, m_threadCondition(AutomaticThreadCondition::create())
- , m_threadLocalCacheLayout(std::make_unique<ThreadLocalCacheLayout>())
{
m_worldState.store(0);
diff --git a/Source/JavaScriptCore/heap/Heap.h b/Source/JavaScriptCore/heap/Heap.h
index 4ce918f..aa374cb 100644
--- a/Source/JavaScriptCore/heap/Heap.h
+++ b/Source/JavaScriptCore/heap/Heap.h
@@ -84,7 +84,6 @@
class SpaceTimeMutatorScheduler;
class StopIfNecessaryTimer;
class SweepingScope;
-class ThreadLocalCacheLayout;
class VM;
class WeakGCMapBase;
struct CurrentThreadState;
@@ -381,8 +380,6 @@
template<typename Func>
void forEachSlotVisitor(const Func&);
- ThreadLocalCacheLayout& threadLocalCacheLayout() { return *m_threadLocalCacheLayout; }
-
Seconds totalGCTime() const { return m_totalGCTime; }
private:
@@ -731,8 +728,6 @@
CurrentThreadState* m_currentThreadState { nullptr };
WTF::Thread* m_currentThread { nullptr }; // It's OK if this becomes a dangling pointer.
-
- std::unique_ptr<ThreadLocalCacheLayout> m_threadLocalCacheLayout;
};
} // namespace JSC
diff --git a/Source/JavaScriptCore/heap/IsoSubspace.cpp b/Source/JavaScriptCore/heap/IsoSubspace.cpp
index 8174b54..8973106 100644
--- a/Source/JavaScriptCore/heap/IsoSubspace.cpp
+++ b/Source/JavaScriptCore/heap/IsoSubspace.cpp
@@ -30,7 +30,6 @@
#include "BlockDirectoryInlines.h"
#include "IsoAlignedMemoryAllocator.h"
#include "LocalAllocatorInlines.h"
-#include "ThreadLocalCacheInlines.h"
namespace JSC {
@@ -38,7 +37,7 @@
: Subspace(name, heap)
, m_size(size)
, m_directory(&heap, WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(size))
- , m_allocator(m_directory.allocator())
+ , m_localAllocator(&m_directory)
, m_isoAlignedMemoryAllocator(std::make_unique<IsoAlignedMemoryAllocator>())
{
initialize(heapCellType, m_isoAlignedMemoryAllocator.get());
@@ -64,10 +63,11 @@
return allocateNonVirtual(vm, size, deferralContext, failureMode);
}
-void* IsoSubspace::allocateNonVirtual(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
+void* IsoSubspace::allocateNonVirtual(VM&, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
{
RELEASE_ASSERT(size == this->size());
- void* result = m_allocator.allocate(vm, deferralContext, failureMode);
+ Allocator allocator = allocatorForNonVirtual(size, AllocatorForMode::MustAlreadyHaveAllocator);
+ void* result = allocator.allocate(deferralContext, failureMode);
return result;
}
diff --git a/Source/JavaScriptCore/heap/IsoSubspace.h b/Source/JavaScriptCore/heap/IsoSubspace.h
index bafe99d..5d7ce59 100644
--- a/Source/JavaScriptCore/heap/IsoSubspace.h
+++ b/Source/JavaScriptCore/heap/IsoSubspace.h
@@ -56,7 +56,7 @@
size_t m_size;
BlockDirectory m_directory;
- Allocator m_allocator;
+ LocalAllocator m_localAllocator;
std::unique_ptr<IsoAlignedMemoryAllocator> m_isoAlignedMemoryAllocator;
SentinelLinkedList<IsoCellSet, BasicRawSentinelNode<IsoCellSet>> m_cellSets;
};
@@ -64,7 +64,7 @@
inline Allocator IsoSubspace::allocatorForNonVirtual(size_t size, AllocatorForMode)
{
RELEASE_ASSERT(size == this->size());
- return m_allocator;
+ return Allocator(&m_localAllocator);
}
#define ISO_SUBSPACE_INIT(heap, heapCellType, type) ("Isolated " #type " Space", (heap), (heapCellType), sizeof(type))
diff --git a/Source/JavaScriptCore/heap/LocalAllocator.cpp b/Source/JavaScriptCore/heap/LocalAllocator.cpp
index 97c7c6e..bfc749f 100644
--- a/Source/JavaScriptCore/heap/LocalAllocator.cpp
+++ b/Source/JavaScriptCore/heap/LocalAllocator.cpp
@@ -32,9 +32,8 @@
namespace JSC {
-LocalAllocator::LocalAllocator(ThreadLocalCache* tlc, BlockDirectory* directory)
- : m_tlc(tlc)
- , m_directory(directory)
+LocalAllocator::LocalAllocator(BlockDirectory* directory)
+ : m_directory(directory)
, m_cellSize(directory->m_cellSize)
, m_freeList(m_cellSize)
{
@@ -42,23 +41,6 @@
directory->m_localAllocators.append(this);
}
-LocalAllocator::LocalAllocator(LocalAllocator&& other)
- : m_tlc(other.m_tlc)
- , m_directory(other.m_directory)
- , m_cellSize(other.m_cellSize)
- , m_freeList(WTFMove(other.m_freeList))
- , m_currentBlock(other.m_currentBlock)
- , m_lastActiveBlock(other.m_lastActiveBlock)
- , m_allocationCursor(other.m_allocationCursor)
-{
- other.reset();
- if (other.isOnList()) {
- auto locker = holdLock(m_directory->m_localAllocatorsLock);
- other.remove();
- m_directory->m_localAllocators.append(this);
- }
-}
-
void LocalAllocator::reset()
{
m_freeList.clear();
@@ -74,25 +56,6 @@
remove();
}
- // Assert that this allocator isn't holding onto any memory. This is a valid assertion for the
- // following two use cases:
- //
- // - Immortal TLC. Those destruct after the heap is done destructing, so they should not have
- // any state left in them.
- //
- // - TLC owned by an object. Such a TLC gets destroyed after a GC flip during which we proved
- // that it is not reachable. Therefore, the TLC should still be in a fully reset state at the
- // time of destruction because for it to get into any other state, someone must have allocated
- // in it (which is impossible because it's supposedly unreachable).
- //
- // My biggest worry with these assertions is that there will be some TLC that gets set as the
- // current one but then never reset, and in the meantime the global object that owns it gets
- // destroyed.
- //
- // Note that if we did hold onto some memory and we wanted to return it then this could be weird.
- // We would potentially have to stopAllocating(). That would mean having to return a block to the
- // BlockDirectory. It's not clear that the BlockDirectory is prepared to handle that during
- // sweeping another block, for example.
bool ok = true;
if (!m_freeList.allocationWillFail()) {
dataLog("FATAL: ", RawPointer(this), "->~LocalAllocator has non-empty free-list.\n");
diff --git a/Source/JavaScriptCore/heap/LocalAllocator.h b/Source/JavaScriptCore/heap/LocalAllocator.h
index 98a260a..9d8f5a4 100644
--- a/Source/JavaScriptCore/heap/LocalAllocator.h
+++ b/Source/JavaScriptCore/heap/LocalAllocator.h
@@ -33,17 +33,17 @@
class BlockDirectory;
class GCDeferralContext;
-class ThreadLocalCache;
class LocalAllocator : public BasicRawSentinelNode<LocalAllocator> {
WTF_MAKE_NONCOPYABLE(LocalAllocator);
public:
- LocalAllocator(ThreadLocalCache*, BlockDirectory*);
- LocalAllocator(LocalAllocator&&);
+ LocalAllocator(BlockDirectory*);
~LocalAllocator();
void* allocate(GCDeferralContext*, AllocationFailureMode);
+
+ unsigned cellSize() const { return m_cellSize; }
void stopAllocating();
void prepareForAllocation();
@@ -55,8 +55,6 @@
bool isFreeListedCell(const void*) const;
- ThreadLocalCache* tlc() const { return m_tlc; }
-
private:
friend class BlockDirectory;
@@ -68,7 +66,6 @@
void* allocateIn(MarkedBlock::Handle*);
ALWAYS_INLINE void doTestCollectionsIfNeeded(GCDeferralContext*);
- ThreadLocalCache* m_tlc;
BlockDirectory* m_directory;
unsigned m_cellSize;
FreeList m_freeList;
diff --git a/Source/JavaScriptCore/heap/ThreadLocalCache.cpp b/Source/JavaScriptCore/heap/ThreadLocalCache.cpp
deleted file mode 100644
index 6fde055..0000000
--- a/Source/JavaScriptCore/heap/ThreadLocalCache.cpp
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Copyright (C) 2018 Apple Inc. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
- * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
- * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
- * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
- * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "config.h"
-#include "ThreadLocalCache.h"
-
-#include "ThreadLocalCacheInlines.h"
-#include "ThreadLocalCacheLayout.h"
-#include <wtf/StdLibExtras.h>
-
-namespace JSC {
-
-RefPtr<ThreadLocalCache> ThreadLocalCache::create(Heap& heap)
-{
- return adoptRef(new ThreadLocalCache(heap));
-}
-
-ThreadLocalCache::ThreadLocalCache(Heap& heap)
- : m_heap(heap)
-{
- m_data = allocateData();
-}
-
-ThreadLocalCache::~ThreadLocalCache()
-{
- destroyData(m_data);
-}
-
-ThreadLocalCache::Data* ThreadLocalCache::allocateData()
-{
- size_t oldSize = m_data ? m_data->size : 0;
- ThreadLocalCacheLayout::Snapshot layout = m_heap.threadLocalCacheLayout().snapshot();
-
- Data* result = static_cast<Data*>(fastMalloc(OBJECT_OFFSETOF(Data, allocator) + layout.size));
- result->size = layout.size;
- result->cache = this;
- for (size_t offset = 0; offset < oldSize; offset += sizeof(LocalAllocator))
- new (&allocator(*result, offset)) LocalAllocator(WTFMove(allocator(*m_data, offset)));
- for (size_t offset = oldSize; offset < layout.size; offset += sizeof(LocalAllocator))
- new (&allocator(*result, offset)) LocalAllocator(this, layout.directories[offset / sizeof(LocalAllocator)]);
- return result;
-}
-
-void ThreadLocalCache::destroyData(Data* data)
-{
- for (size_t offset = 0; offset < data->size; offset += sizeof(LocalAllocator))
- allocator(*data, offset).~LocalAllocator();
- fastFree(data);
-}
-
-void ThreadLocalCache::installSlow(VM& vm, RefPtr<ThreadLocalCache>* previous)
-{
-#if USE(FAST_TLS_FOR_TLC)
- static std::once_flag onceFlag;
- std::call_once(
- onceFlag,
- [] () {
- pthread_key_init_np(tlsKey, destructor);
- });
-#endif
-
- ref();
-
- if (ThreadLocalCache::Data* oldCacheData = getImpl(vm)) {
- ThreadLocalCache* oldCache = oldCacheData->cache;
- if (previous)
- *previous = adoptRef(oldCache);
- else
- oldCache->deref();
- }
-
- installData(vm, m_data);
-}
-
-void ThreadLocalCache::installData(VM& vm, Data* data)
-{
-#if USE(FAST_TLS_FOR_TLC)
- UNUSED_PARAM(vm);
- _pthread_setspecific_direct(tlsKey, data);
-#else
- vm.threadLocalCacheData = data;
-#endif
-}
-
-LocalAllocator& ThreadLocalCache::allocatorSlow(VM& vm, size_t offset)
-{
- Data* oldData = m_data;
- m_data = allocateData();
- destroyData(oldData);
- installData(vm, m_data);
- RELEASE_ASSERT(offset < m_data->size);
- return allocator(*m_data, offset);
-}
-
-void ThreadLocalCache::destructor(void* arg)
-{
- if (!arg)
- return;
- Data* data = static_cast<Data*>(arg);
- data->cache->deref();
-}
-
-} // namespace JSC
-
diff --git a/Source/JavaScriptCore/heap/ThreadLocalCache.h b/Source/JavaScriptCore/heap/ThreadLocalCache.h
deleted file mode 100644
index e8842ec..0000000
--- a/Source/JavaScriptCore/heap/ThreadLocalCache.h
+++ /dev/null
@@ -1,105 +0,0 @@
-/*
- * Copyright (C) 2018 Apple Inc. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
- * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
- * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
- * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
- * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#pragma once
-
-#include "AllocationFailureMode.h"
-#include "LocalAllocator.h"
-#include <wtf/FastMalloc.h>
-#include <wtf/FastTLS.h>
-#include <wtf/ThreadSafeRefCounted.h>
-#include <wtf/Vector.h>
-
-namespace JSC {
-
-class Heap;
-class VM;
-
-class ThreadLocalCache : public ThreadSafeRefCounted<ThreadLocalCache> {
- WTF_MAKE_NONCOPYABLE(ThreadLocalCache);
- WTF_MAKE_FAST_ALLOCATED;
-
-public:
- JS_EXPORT_PRIVATE static RefPtr<ThreadLocalCache> create(Heap&);
-
- JS_EXPORT_PRIVATE virtual ~ThreadLocalCache();
-
- static RefPtr<ThreadLocalCache> get(VM&);
-
- // This is designed to be fast enough that you could even call it before every allocation, by
- // optimizing for the case that you're just installing the cache that is already installed. This
- // assumes a relatively small number of caches or low chance of actual context switch combined
- // with possibly high rate of "I may have context switched" sites that call this out of paranoia.
- void install(VM&, RefPtr<ThreadLocalCache>* = nullptr);
-
- static LocalAllocator& allocator(VM&, size_t offset);
-
- template<typename SuccessFunc, typename FailureFunc>
- static void tryGetAllocator(VM&, size_t offset, const SuccessFunc&, const FailureFunc&);
-
- static ptrdiff_t offsetOfSizeInData() { return OBJECT_OFFSETOF(Data, size); }
- static ptrdiff_t offsetOfFirstAllocatorInData() { return OBJECT_OFFSETOF(Data, allocator); }
-
-protected:
- JS_EXPORT_PRIVATE ThreadLocalCache(Heap&);
-
-private:
- friend class VM;
-
- struct Data {
- size_t size;
- ThreadLocalCache* cache;
- LocalAllocator allocator[1];
- };
-
- static Data* getImpl(VM&);
-
- Data* allocateData();
- void destroyData(Data*);
- static LocalAllocator& allocator(Data& data, size_t offset);
-
- void installSlow(VM&, RefPtr<ThreadLocalCache>*);
- static void installData(VM&, Data*);
-
- LocalAllocator& allocatorSlow(VM&, size_t offset);
-
- static void destructor(void*);
-
- // When installed, we ref() the cache. Uninstalling deref()s the cache. TLS destruction deref()s
- // the cache. When the cache is destructed, it needs to return all of its state to the GC. I
- // think that just means stopAllocating(), but with some seriously heavy caveats having to do
- // with when it's valid to make that call. Alternatively, we could RELEASE_ASSERT that the cache
- // is empty when we destruct it. This is correct for caches that are kept live by GC objects and
- // it's correct for immortal caches.
- Heap& m_heap;
- Data* m_data { nullptr };
-
-#if USE(FAST_TLS_FOR_TLC)
- static const pthread_key_t tlsKey = WTF_GC_TLC_KEY;
-#endif
-};
-
-} // namespace JSC
-
diff --git a/Source/JavaScriptCore/heap/ThreadLocalCacheInlines.h b/Source/JavaScriptCore/heap/ThreadLocalCacheInlines.h
deleted file mode 100644
index b64f978..0000000
--- a/Source/JavaScriptCore/heap/ThreadLocalCacheInlines.h
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Copyright (C) 2018 Apple Inc. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
- * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
- * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
- * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
- * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#pragma once
-
-#include "ThreadLocalCache.h"
-#include "VM.h"
-
-namespace JSC {
-
-inline ThreadLocalCache::Data* ThreadLocalCache::getImpl(VM& vm)
-{
-#if USE(FAST_TLS_FOR_TLC)
- UNUSED_PARAM(vm);
- return static_cast<Data*>(_pthread_getspecific_direct(tlsKey));
-#else
- return vm.threadLocalCacheData;
-#endif
-}
-
-inline RefPtr<ThreadLocalCache> ThreadLocalCache::get(VM& vm)
-{
- ThreadLocalCache::Data* data = getImpl(vm);
- if (LIKELY(data))
- return data->cache;
- return nullptr;
-}
-
-inline void ThreadLocalCache::install(VM& vm, RefPtr<ThreadLocalCache>* previous)
-{
- if (getImpl(vm) == m_data) {
- if (previous)
- *previous = nullptr;
- return;
- }
- installSlow(vm, previous);
-}
-
-inline LocalAllocator& ThreadLocalCache::allocator(VM& vm, size_t offset)
-{
- ThreadLocalCache::Data* data = getImpl(vm);
- if (LIKELY(offset < data->size))
- return allocator(*data, offset);
- return data->cache->allocatorSlow(vm, offset);
-}
-
-template<typename SuccessFunc, typename FailureFunc>
-void ThreadLocalCache::tryGetAllocator(VM& vm, size_t offset, const SuccessFunc& successFunc, const FailureFunc& failureFunc)
-{
- ThreadLocalCache::Data* data = getImpl(vm);
- if (LIKELY(offset < data->size))
- successFunc(allocator(*data, offset));
- else
- failureFunc();
-}
-
-inline LocalAllocator& ThreadLocalCache::allocator(Data& data, size_t offset)
-{
- return *bitwise_cast<LocalAllocator*>(bitwise_cast<char*>(&data.allocator[0]) + offset);
-}
-
-} // namespace JSC
-
diff --git a/Source/JavaScriptCore/heap/ThreadLocalCacheLayout.cpp b/Source/JavaScriptCore/heap/ThreadLocalCacheLayout.cpp
deleted file mode 100644
index c1ac87f..0000000
--- a/Source/JavaScriptCore/heap/ThreadLocalCacheLayout.cpp
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Copyright (C) 2018 Apple Inc. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
- * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
- * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
- * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
- * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "config.h"
-#include "ThreadLocalCacheLayout.h"
-
-#include "BlockDirectory.h"
-
-namespace JSC {
-
-ThreadLocalCacheLayout::ThreadLocalCacheLayout()
-{
-}
-
-ThreadLocalCacheLayout::~ThreadLocalCacheLayout()
-{
-}
-
-void ThreadLocalCacheLayout::allocateOffset(BlockDirectory* directory)
-{
- auto locker = holdLock(m_lock);
- directory->m_tlcOffset = m_size;
- m_size += sizeof(LocalAllocator);
- m_directories.append(directory);
-}
-
-ThreadLocalCacheLayout::Snapshot ThreadLocalCacheLayout::snapshot()
-{
- auto locker = holdLock(m_lock);
- Snapshot result;
- result.size = m_size;
- result.directories = m_directories;
- return result;
-}
-
-BlockDirectory* ThreadLocalCacheLayout::directory(unsigned offset)
-{
- auto locker = holdLock(m_lock);
- return m_directories[offset / sizeof(LocalAllocator)];
-}
-
-} // namespace JSC
-
diff --git a/Source/JavaScriptCore/heap/ThreadLocalCacheLayout.h b/Source/JavaScriptCore/heap/ThreadLocalCacheLayout.h
deleted file mode 100644
index e5ea1b6..0000000
--- a/Source/JavaScriptCore/heap/ThreadLocalCacheLayout.h
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Copyright (C) 2018 Apple Inc. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
- * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
- * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
- * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
- * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#pragma once
-
-#include <wtf/FastMalloc.h>
-#include <wtf/Lock.h>
-#include <wtf/Noncopyable.h>
-#include <wtf/Vector.h>
-
-namespace JSC {
-
-class BlockDirectory;
-
-// Each Heap has a ThreadLocalCacheLayout that helps us figure out how to allocate a ThreadLocalCache
-// for that Heap.
-
-class ThreadLocalCacheLayout {
- WTF_MAKE_NONCOPYABLE(ThreadLocalCacheLayout);
- WTF_MAKE_FAST_ALLOCATED;
-
-public:
- ThreadLocalCacheLayout();
- ~ThreadLocalCacheLayout();
-
- // BlockDirectory calls this during creation and this fills in BlockDirectory::offset.
- void allocateOffset(BlockDirectory*);
-
- struct Snapshot {
- size_t size;
- Vector<BlockDirectory*> directories;
- };
-
- Snapshot snapshot();
-
- BlockDirectory* directory(unsigned offset);
-
-private:
- Lock m_lock;
- size_t m_size { 0 };
- Vector<BlockDirectory*> m_directories;
-};
-
-} // namespace JSC
-
diff --git a/Source/JavaScriptCore/jit/AssemblyHelpers.cpp b/Source/JavaScriptCore/jit/AssemblyHelpers.cpp
index 8529dda..6cca650 100644
--- a/Source/JavaScriptCore/jit/AssemblyHelpers.cpp
+++ b/Source/JavaScriptCore/jit/AssemblyHelpers.cpp
@@ -588,24 +588,13 @@
Jump popPath;
Jump done;
-#if USE(FAST_TLS_FOR_TLC)
- loadFromTLSPtr(fastTLSOffsetForKey(WTF_GC_TLC_KEY), scratchGPR);
-#else
- loadPtr(&vm().threadLocalCacheData, scratchGPR);
-#endif
- if (allocator.isConstant()) {
- slowPath.append(branch32(BelowOrEqual, Address(scratchGPR, ThreadLocalCache::offsetOfSizeInData()), TrustedImm32(allocator.allocator().offset())));
- addPtr(TrustedImm32(ThreadLocalCache::offsetOfFirstAllocatorInData() + allocator.allocator().offset()), scratchGPR, allocatorGPR);
- } else {
- slowPath.append(branch32(BelowOrEqual, Address(scratchGPR, ThreadLocalCache::offsetOfSizeInData()), allocatorGPR));
- addPtr(TrustedImm32(ThreadLocalCache::offsetOfFirstAllocatorInData()), allocatorGPR);
- addPtr(scratchGPR, allocatorGPR);
- }
+ if (allocator.isConstant())
+ move(TrustedImmPtr(allocator.allocator().localAllocator()), allocatorGPR);
load32(Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfRemaining()), resultGPR);
popPath = branchTest32(Zero, resultGPR);
if (allocator.isConstant())
- add32(TrustedImm32(-allocator.allocator().cellSize(vm().heap)), resultGPR, scratchGPR);
+ add32(TrustedImm32(-allocator.allocator().cellSize()), resultGPR, scratchGPR);
else {
move(resultGPR, scratchGPR);
sub32(Address(allocatorGPR, LocalAllocator::offsetOfCellSize()), scratchGPR);
@@ -638,7 +627,8 @@
slowPath.append(jump());
return;
}
- }
+ } else
+ slowPath.append(branchTestPtr(Zero, allocatorGPR));
emitAllocateWithNonNullAllocator(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
}
@@ -652,7 +642,7 @@
urshift32(TrustedImm32(stepShift), scratchGPR1);
slowPath.append(branch32(Above, scratchGPR1, TrustedImm32(MarkedSpace::largeCutoff >> stepShift)));
move(TrustedImmPtr(subspace.allocatorForSizeStep() - 1), scratchGPR2);
- load32(BaseIndex(scratchGPR2, scratchGPR1, TimesFour), scratchGPR1);
+ loadPtr(BaseIndex(scratchGPR2, scratchGPR1, timesPtr()), scratchGPR1);
emitAllocate(resultGPR, JITAllocator::variable(), scratchGPR1, scratchGPR2, slowPath);
}
diff --git a/Source/JavaScriptCore/jit/JITOpcodes.cpp b/Source/JavaScriptCore/jit/JITOpcodes.cpp
index c6f1258..be8b15b 100644
--- a/Source/JavaScriptCore/jit/JITOpcodes.cpp
+++ b/Source/JavaScriptCore/jit/JITOpcodes.cpp
@@ -834,9 +834,8 @@
loadPtr(Address(calleeReg, JSFunction::offsetOfRareData()), rareDataReg);
addSlowCase(branchTestPtr(Zero, rareDataReg));
xorPtr(TrustedImmPtr(JSFunctionPoison::key()), rareDataReg);
- load32(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorReg);
+ loadPtr(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorReg);
loadPtr(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureReg);
- addSlowCase(branch32(Equal, allocatorReg, TrustedImm32(Allocator().offset())));
loadPtr(cachedFunction, cachedFunctionReg);
Jump hasSeenMultipleCallees = branchPtr(Equal, cachedFunctionReg, TrustedImmPtr(JSCell::seenMultipleCalleeObjects()));
diff --git a/Source/JavaScriptCore/jit/JITOpcodes32_64.cpp b/Source/JavaScriptCore/jit/JITOpcodes32_64.cpp
index 44955e3..9133abc 100644
--- a/Source/JavaScriptCore/jit/JITOpcodes32_64.cpp
+++ b/Source/JavaScriptCore/jit/JITOpcodes32_64.cpp
@@ -957,7 +957,6 @@
addSlowCase(branchTestPtr(Zero, rareDataReg));
load32(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorReg);
loadPtr(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureReg);
- addSlowCase(branch32(Equal, allocatorReg, TrustedImm32(Allocator().offset())));
loadPtr(cachedFunction, cachedFunctionReg);
Jump hasSeenMultipleCallees = branchPtr(Equal, cachedFunctionReg, TrustedImmPtr(JSCell::seenMultipleCalleeObjects()));
diff --git a/Source/JavaScriptCore/runtime/JSLock.cpp b/Source/JavaScriptCore/runtime/JSLock.cpp
index f948e88..dfb75a7 100644
--- a/Source/JavaScriptCore/runtime/JSLock.cpp
+++ b/Source/JavaScriptCore/runtime/JSLock.cpp
@@ -28,7 +28,6 @@
#include "JSCInlines.h"
#include "MachineStackMarker.h"
#include "SamplingProfiler.h"
-#include "ThreadLocalCacheInlines.h"
#include "WasmMachineThreads.h"
#include <thread>
#include <wtf/Threading.h>
@@ -149,8 +148,6 @@
m_vm->setLastStackTop(thread.savedLastStackTop());
ASSERT(thread.stack().contains(m_vm->lastStackTop()));
- m_vm->defaultThreadLocalCache->install(*m_vm);
-
m_vm->heap.machineThreads().addCurrentThread();
#if ENABLE(WEBASSEMBLY)
Wasm::startTrackingCurrentThread();
diff --git a/Source/JavaScriptCore/runtime/VM.cpp b/Source/JavaScriptCore/runtime/VM.cpp
index 89d0f30..2af9d0a 100644
--- a/Source/JavaScriptCore/runtime/VM.cpp
+++ b/Source/JavaScriptCore/runtime/VM.cpp
@@ -138,7 +138,6 @@
#include "StrongInlines.h"
#include "StructureInlines.h"
#include "TestRunnerUtils.h"
-#include "ThreadLocalCacheInlines.h"
#include "ThunkGenerators.h"
#include "TypeProfiler.h"
#include "TypeProfilerLog.h"
@@ -355,9 +354,6 @@
updateSoftReservedZoneSize(Options::softReservedZoneSize());
setLastStackTop(stack.origin());
- defaultThreadLocalCache = ThreadLocalCache::create(heap);
- defaultThreadLocalCache->install(*this);
-
// Need to be careful to keep everything consistent here
JSLockHolder lock(this);
AtomicStringTable* existingEntryAtomicStringTable = Thread::current().setCurrentAtomicStringTable(m_atomicStringTable);
@@ -559,10 +555,6 @@
m_apiLock->willDestroyVM(this);
heap.lastChanceToFinalize();
-#if !USE(FAST_TLS_FOR_TLC)
- ThreadLocalCache::destructor(threadLocalCacheData);
-#endif
-
delete interpreter;
#ifndef NDEBUG
interpreter = reinterpret_cast<Interpreter*>(0xbbadbeef);
diff --git a/Source/JavaScriptCore/runtime/VM.h b/Source/JavaScriptCore/runtime/VM.h
index cb52270..c815ad4 100644
--- a/Source/JavaScriptCore/runtime/VM.h
+++ b/Source/JavaScriptCore/runtime/VM.h
@@ -53,7 +53,6 @@
#include "StructureCache.h"
#include "VMEntryRecord.h"
#include "VMTraps.h"
-#include "ThreadLocalCache.h"
#include "WasmContext.h"
#include "Watchpoint.h"
#include <wtf/BumpPointerAllocator.h>
@@ -688,11 +687,6 @@
JSObject* stringRecursionCheckFirstObject { nullptr };
HashSet<JSObject*> stringRecursionCheckVisitedObjects;
-#if !USE(FAST_TLS_FOR_TLC)
- ThreadLocalCache::Data* threadLocalCacheData { nullptr };
-#endif
- RefPtr<ThreadLocalCache> defaultThreadLocalCache;
-
LocalTimeOffsetCache localTimeOffsetCache;
String cachedDateString;
diff --git a/Source/JavaScriptCore/runtime/VMEntryScope.cpp b/Source/JavaScriptCore/runtime/VMEntryScope.cpp
index ba15199..41a81fc 100644
--- a/Source/JavaScriptCore/runtime/VMEntryScope.cpp
+++ b/Source/JavaScriptCore/runtime/VMEntryScope.cpp
@@ -30,7 +30,6 @@
#include "JSGlobalObject.h"
#include "Options.h"
#include "SamplingProfiler.h"
-#include "ThreadLocalCacheInlines.h"
#include "VM.h"
#include "Watchdog.h"
#include <wtf/StackBounds.h>
@@ -71,9 +70,6 @@
VMEntryScope::~VMEntryScope()
{
- if (m_previousTLC)
- m_previousTLC->install(m_vm);
-
if (m_vm.entryScope != this)
return;
diff --git a/Source/JavaScriptCore/runtime/VMEntryScope.h b/Source/JavaScriptCore/runtime/VMEntryScope.h
index 2993b3d..c2971f6 100644
--- a/Source/JavaScriptCore/runtime/VMEntryScope.h
+++ b/Source/JavaScriptCore/runtime/VMEntryScope.h
@@ -31,7 +31,6 @@
namespace JSC {
class JSGlobalObject;
-class ThreadLocalCache;
class VM;
class VMEntryScope {
@@ -48,7 +47,6 @@
VM& m_vm;
JSGlobalObject* m_globalObject;
Vector<std::function<void ()>> m_didPopListeners;
- RefPtr<ThreadLocalCache> m_previousTLC;
};
} // namespace JSC
diff --git a/Source/WTF/ChangeLog b/Source/WTF/ChangeLog
index 05b0a5a..2c565e4 100644
--- a/Source/WTF/ChangeLog
+++ b/Source/WTF/ChangeLog
@@ -1,3 +1,12 @@
+2018-05-22 Filip Pizlo <fpizlo@apple.com>
+
+ Get rid of TLCs
+ https://bugs.webkit.org/show_bug.cgi?id=185846
+
+ Rubber stamped by Geoffrey Garen.
+
+ * wtf/Platform.h:
+
2018-05-22 Ryan Haddad <ryanhaddad@apple.com>
Unreviewed, rolling out r232052.
diff --git a/Source/WTF/wtf/Platform.h b/Source/WTF/wtf/Platform.h
index 0fa30be..c7d0ca0 100644
--- a/Source/WTF/wtf/Platform.h
+++ b/Source/WTF/wtf/Platform.h
@@ -793,11 +793,6 @@
#define ENABLE_FAST_TLS_JIT 1
#endif
-/* This feature is currently disabled because WebCore will context switch VMs without telling JSC.
- FIXME: Re-enable this feature.
- https://bugs.webkit.org/show_bug.cgi?id=182173 */
-#define USE_FAST_TLS_FOR_TLC 0
-
#if CPU(X86) || CPU(X86_64) || CPU(ARM_THUMB2) || CPU(ARM64) || CPU(ARM_TRADITIONAL) || CPU(MIPS)
#define ENABLE_MASM_PROBE 1
#else
