elasticsearch - Facing memory issues with elastic search while indexing -


i have 3 nodes(aws elasticsearch service) of 16gb running elastic search service. have configured es have 3 shards 2 replicas. these 3 nodes responsible indexing of data , querying , fetching of data.

heap allocation 3 7gb.

data split 2 indices, index1 , index2. index1 not have data , document sizes small. index2 heavier index max document size in 1mb.

however, many times while indexing documents index2, 1 of es instance throws outofmemoryexception. checked shards memory , there total of 3.5 gb of document data in index2.

i unable figure out cause , looking in debugging issue.

the structure of document similar : 

    {           "name":"abc",         "class":10,         "school":"xyz",         "subjects":[                  {                     "a":{                     "name":"subject a",                   "marks":80,                   "passed":true                   }                },                {                    "b":{                      "name":"subject b",                    "marks":76,                    "passed":true                }             }         ]       }

the array in "subjects" key can go upto length of 3000.

the es version being used 5.1.0.

find stack trace below : 

[2017-07-28t15:44:13,912][warn ][o.e.m.j.jvmgcmonitorservice] [oblzpwi] [gc][849973] overhead, spent [26s] collecting in last [26.1s] [2017-07-28t15:44:13,895][warn ][o.e.t.n.netty4transport  ] [oblzpwi] exception caught on transport layer [[!!!io.netty.channel.socket.nio.niosocketchannel@650b7925=>java.lang.outofmemoryerror:compressed class space!!!]], closing connection org.elasticsearch.elasticsearchexception: java.lang.outofmemoryerror: compressed class space     @ org.elasticsearch.transport.netty4.netty4transport.exceptioncaught(netty4transport.java:326) [transport-netty4-5.1.2.jar:5.1.2]     @ org.elasticsearch.transport.netty4.netty4messagechannelhandler.exceptioncaught(netty4messagechannelhandler.java:84) [transport-netty4-5.1.2.jar:5.1.2]     @ io.netty.channel.abstractchannelhandlercontext.invokeexceptioncaught(abstractchannelhandlercontext.java:296) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.notifyhandlerexception(abstractchannelhandlercontext.java:861) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:375) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:359) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.firechannelread(abstractchannelhandlercontext.java:351) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.handler.codec.bytetomessagedecoder.firechannelread(bytetomessagedecoder.java:293) [netty-codec-4.1.6.final.jar:4.1.6.final]     @ io.netty.handler.codec.bytetomessagedecoder.firechannelread(bytetomessagedecoder.java:280) [netty-codec-4.1.6.final.jar:4.1.6.final]     @ io.netty.handler.codec.bytetomessagedecoder.calldecode(bytetomessagedecoder.java:396) [netty-codec-4.1.6.final.jar:4.1.6.final]     @ io.netty.handler.codec.bytetomessagedecoder.channelread(bytetomessagedecoder.java:248) [netty-codec-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:373) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:359) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.firechannelread(abstractchannelhandlercontext.java:351) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.channelinboundhandleradapter.channelread(channelinboundhandleradapter.java:86) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:373) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:359) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.firechannelread(abstractchannelhandlercontext.java:351) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.defaultchannelpipeline$headcontext.channelread(defaultchannelpipeline.java:1334) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:373) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.abstractchannelhandlercontext.invokechannelread(abstractchannelhandlercontext.java:359) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.defaultchannelpipeline.firechannelread(defaultchannelpipeline.java:926) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.nio.abstractniobytechannel$niobyteunsafe.read(abstractniobytechannel.java:129) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.nio.nioeventloop.processselectedkey(nioeventloop.java:651) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.nio.nioeventloop.processselectedkeysplain(nioeventloop.java:536) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.nio.nioeventloop.processselectedkeys(nioeventloop.java:490) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.channel.nio.nioeventloop.run(nioeventloop.java:450) [netty-transport-4.1.6.final.jar:4.1.6.final]     @ io.netty.util.concurrent.singlethreadeventexecutor$5.run(singlethreadeventexecutor.java:873) [netty-common-4.1.6.final.jar:4.1.6.final]     @ java.lang.thread.run(thread.java:748) [?:1.8.0_131] caused by: java.lang.outofmemoryerror: compressed class space

you're running out of compressed class space, rather heap memory. can configured using -xx: compressedclassspacesize=1g (as example value).

you can read more on oracle's docs here, second bottom.

however, may treating symptom.

there's known problems older versions of elastic described on github issue. specific issue caused compiling unique script each request. if sounds relevant problem, might worth updating answer actual api calls trigger exception.


Comments

Post a Comment

Popular posts from this blog

javascript - Create a stacked percentage column -

i have collection of data : var data = [ {"p301a":"10","p301b":"7","p301c":"7","p301d":"3","p301e":"8","p301f":"8","p301g":"8","p301h":"8","p301i":"8","p301j":"8","p301k":"8","p301l":"8","p301m":"8","p301n":"8","p301o":"8","age":"31-40 years","profesion":"2","position":"2"}, {"p301a":"5","p301b":"4","p301c":"4","p301d":"4","p301e":"4","p301f":"4","p301g":"4","p301h":"4","p301i":"4","p301j":"4","p301k":"4...
Read more

Optimising Firebase database by automatically overwriting data -

in firebase app have following structure: posts latest all users can write posts > latest . want limit entries 20 posts save space in database. , posts beyond 20 unnecessary never shown on homepage. how can limit posts 20, when user writes posts > latest posts @ end drop off (be deleted) automatically? this sample should you. you need : const max_log_count = 20; exports.removeold = functions.database.ref('/posts/latest/{postid}').oncreate(event => { const parentref = event.data.ref.parent; return parentref.once('value').then(snapshot => { if (snapshot.numchildren() >= max_log_count) { let childcount = 0; const updates = {}; snapshot.foreach(function(child) { if (++childcount <= snapshot.numchildren() - max_log_count) { updates[child.key] = null; } }); // update parent. removes children. ...
Read more

javascript - Angular UI-Grid customTemplate directive causing rows to load slowly/? -

Image
each grid cell can have more 1 customtemplate loaded. have directive holds watcher on model_col_field listen cell updates on scroll when dom rows updated , recompile celltemplate . without watcher , $compile rows repeat on scroll, showing same set of rows. is there more angular way of approaching or way improve performance? relevant controller code: create gridoptions.data & gridoptions.columndefs : var customtemp = '<div ng-repeat="opt in grid.appscope.colvalueoptionsmodel" ng-init="tempstring=model_col_field.clazz+opt.key;"><comparison-directive dbo="model_col_field" attobj="opt" fff="grid.appscope.getattributetemplate(tempstring)" /></div>'; $scope.gridoptions.columndefs = [ { name:'yindex',pinnedleft:true,celltemplate: customtemp, displayname:'index'}, { name:'col1',celltemplate: customtemp}, { name:'col2',celltemplate: c...
Read more