Profile Image
kwh062793

Why is my opensearch server taking so long to complete the ssl handshake?

Intermittently my opensearch takes a very long time to ssl handshake.

 

$ curl -X GET https://hostname:9200/_prometheus/metrics -I -u prometheus -k -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" -s

 

HTTP/1.1 200 OK
content-type: text/plain; charset=UTF-8
content-length: 77762

time_namelookup: 0.002251
time_connect: 0.009924
time_appconnect: 7.167518
time_pretransfer: 7.167673
time_redirect: 0.000000
time_starttransfer: 7.711578
time_total: 7.711712

 

"time_appconnect" is time to complete ssl handshake

 

The statistical results of strace are as follows.

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 50.06 4131.265271      102772     40198     10050 futex
 28.25 2331.284698       78908     29544           epoll_wait
  7.99  659.105023        7695     85643           write
  6.25  516.141664     5376475        96        17 restart_syscall
  4.04  333.149867        8186     40695           read
  0.69   56.831350        7272      7815        84 stat
  0.49   40.848041        6609      6180           getrusage
  0.40   32.975234        7065      4667           fstat
  0.35   29.182887        7332      3980           fdatasync
  0.27   22.582614        7382      3059           open
  0.26   21.469858        7297      2942           close
  0.20   16.612980        4456      3728           gettid
  0.19   15.827820        6917      2288           pread64
  0.18   14.586857        7330      1990           pwrite64
  0.10    8.113931        7105      1142        22 lstat
  0.08    6.297281        6837       921           unlink
  0.07    5.608822        7154       784           sched_yield
  0.04    3.512373        6421       547           mmap
  0.04    3.318214        6320       525           munmap
  0.01    0.827707        8277       100           clock_gettime
  0.01    0.689399        7034        98           fcntl
  0.01    0.675389        7341        92           getdents
  0.00    0.374128        6563        57           rt_sigprocmask
  0.00    0.334623        7274        46           dup
  0.00    0.220867        5258        42        18 sched_getaffinity
  0.00    0.209089        8363        25           fsync
  0.00    0.123429        4408        28           rt_sigreturn
  0.00    0.117502        3263        36           nanosleep
  0.00    0.111040       12337         9           clone
  0.00    0.104197        4007        26           sysinfo
  0.00    0.101055        8421        12           rename
  0.00    0.098035        8912        11           access
  0.00    0.061713        5142        12           mprotect
  0.00    0.058096        9682         6           lseek
  0.00    0.055275        6141         9           set_robust_list
  0.00    0.053153        6644         8           prctl
  0.00    0.019488        2436         8           setsockopt
  0.00    0.019141        2392         8           getsockopt
  0.00    0.018171        9085         2           sendfile
  0.00    0.016803        5601         3           epoll_ctl
  0.00    0.014648        7323         2           madvise
  0.00    0.008149        2037         4           getsockname
  0.00    0.005010        2504         2           accept
  0.00    0.002537        2537         1         1 shutdown
  0.00    0.001630         135        12           times
------ ----------- ----------- --------- --------- ----------------
100.00 8253.035059                237403     10192 total

 

my opensearch version is 2.3.0 and cluster status are as follows

{
  "cluster_name": "my_cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 275,
  "number_of_data_nodes": 270,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 18071,
  "active_shards": 36411,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100.0
}


Report URL - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjUvMDMvMTQvdGhyZWFkZHVtcC50eHQtLTEyLTEwLTU3

    Please Sign In or to post your comment or answer

    Profile Image

    Kousika M

    Hello kwh062793,


    Greetings!

    Your OpenSearch server is taking a long time to complete the SSL handshake, which might be related to high CPU usage. Our application detected that certain threads are consuming a significant amount of CPU. This could slow down SSL processing and other operations. Please review the CPU-consuming threads section to investigate further. I have attached the screenshot for reference.



    Please find the stacktraces of these threads consuming high CPU. This will help,

    opensearch[pgsca2x1064.nfra.io][transport_worker][T#139]

thread Id : opensearch[pgsca2x1064.nfra.io][transport_worker][T#139]

state : RUNNABLE

stackTrace:
java.lang.Thread.State: RUNNABLE
at java.base@17.0.4/java.util.zip.Inflater.inflateBytesBytes(Native Method)
at java.base@17.0.4/java.util.zip.Inflater.inflate(Inflater.java:378)
- locked <79ac5391> (a java.util.zip.Inflater$InflaterZStreamRef)
at app//org.opensearch.transport.TransportDecompressor.decompress(TransportDecompressor.java:110)
at app//org.opensearch.transport.InboundDecoder.decompress(InboundDecoder.java:156)
at app//org.opensearch.transport.InboundDecoder.internalDecode(InboundDecoder.java:120)
at app//org.opensearch.transport.InboundDecoder.decode(InboundDecoder.java:71)
at app//org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:134)
at app//org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115)
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base@17.0.4/java.lang.Thread.run(Thread.java:833)
Locked ownable synchronizers:
- None

     

    opensearch[pgsca2x1064.nfra.io][http_server_worker][T#9]


thread Id : opensearch[pgsca2x1064.nfra.io][http_server_worker][T#9]

state : RUNNABLE


stackTrace:
java.lang.Thread.State: RUNNABLE
at app//org.apache.logging.log4j.message.ParameterFormatter.countArgumentPlaceholders2(ParameterFormatter.java:113)
at app//org.apache.logging.log4j.message.ParameterizedMessage.init(ParameterizedMessage.java:149)
at app//org.apache.logging.log4j.message.ParameterizedMessage.<init>(ParameterizedMessage.java:123)
at app//org.opensearch.common.logging.OpenSearchLogMessage.<init>(OpenSearchLogMessage.java:56)
at app//org.opensearch.common.logging.DeprecatedMessage.<init>(DeprecatedMessage.java:55)
at app//org.opensearch.common.logging.DeprecationLogger$DeprecationLoggerBuilder.withDeprecation(DeprecationLogger.java:116)
at app//org.opensearch.common.logging.DeprecationLogger.deprecate(DeprecationLogger.java:105)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.lambda$checkSystemIndexAccess$3(IndexNameExpressionResolver.java:374)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver$$Lambda$3986/0x0000000801957bd0.accept(Unknown Source)
at java.base@17.0.4/java.util.ArrayList.forEach(ArrayList.java:1511)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.checkSystemIndexAccess(IndexNameExpressionResolver.java:373)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:360)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:237)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:159)
at org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.getResolvedIndexPattern(ConfigModelV7.java:778)
at org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.attemptResolveIndexNames(ConfigModelV7.java:763)
at org.opensearch.security.securityconf.ConfigModelV7.lambda$impliesTypePerm$4(ConfigModelV7.java:1011)
at org.opensearch.security.securityconf.ConfigModelV7$$Lambda$5647/0x0000000801c0ef08.apply(Unknown Source)
at java.base@17.0.4/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base@17.0.4/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
at java.base@17.0.4/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base@17.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base@17.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base@17.0.4/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base@17.0.4/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at org.opensearch.security.securityconf.ConfigModelV7.impliesTypePerm(ConfigModelV7.java:1012)
at org.opensearch.security.securityconf.ConfigModelV7$SecurityRoles.impliesTypePermGlobal(ConfigModelV7.java:489)
at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:460)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:303)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
at app//org.opensearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:632)
at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at app//org.opensearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:774)
at app//org.opensearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:309)
at app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:220)
at app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:118)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:317)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
at app//org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
at app//org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
at app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:426)
at app//org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:502)
at app//org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111)
at app//org.opensearch.rest.action.document.RestBulkAction$$Lambda$5593/0x0000000801c12170.accept(Unknown Source)
at app//org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125)
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:127)
at app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:312)
at app//org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398)
at app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:241)
at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
at app//org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366)
at app//org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445)
at app//org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356)
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55)
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base@17.0.4/java.lang.Thread.run(Thread.java:833)
Locked ownable synchronizers:
- None

     

    opensearch[pgsca2x1064.nfra.io][search][T#60]


thread Id : opensearch[pgsca2x1064.nfra.io][search][T#60]

state : RUNNABLE


stackTrace:
java.lang.Thread.State: RUNNABLE
at app//org.apache.lucene.index.OrdinalMap.<init>(OrdinalMap.java:299)
at app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:182)
at app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:160)
at app//org.opensearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:83)
at app//org.opensearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:169)
at app//org.opensearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:62)
at app//org.opensearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:205)
at app//org.opensearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$5508/0x0000000801bf1a28.load(Unknown Source)
at app//org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:461)
at app//org.opensearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:202)
at app//org.opensearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalInternal(AbstractIndexOrdinalsFieldData.java:157)
at app//org.opensearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobal(AbstractIndexOrdinalsFieldData.java:123)
at app//org.opensearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:256)
at app//org.opensearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:224)
at app//org.opensearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:362)
at app//org.opensearch.search.aggregations.bucket.terms.TermsAggregatorFactory$1.build(TermsAggregatorFactory.java:123)
at app//org.opensearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:311)
at app//org.opensearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:76)
at app//org.opensearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:101)
at app//org.opensearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:278)
at app//org.opensearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:68)
at app//org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:151)
at app//org.opensearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1532)
at app//org.opensearch.indices.IndicesService$$Lambda$4918/0x0000000801b4d4e0.accept(Unknown Source)
at app//org.opensearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1591)
at app//org.opensearch.indices.IndicesService$$Lambda$4919/0x0000000801b4dc40.get(Unknown Source)
at app//org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:201)
at app//org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:184)
at app//org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:461)
at app//org.opensearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:151)
at app//org.opensearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1597)
at app//org.opensearch.indices.IndicesService.loadIntoContext(IndicesService.java:1531)
at app//org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:448)
at app//org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:514)
at app//org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:483)
at app//org.opensearch.search.SearchService$2$$Lambda$4893/0x0000000801b37818.get(Unknown Source)
at app//org.opensearch.search.SearchService$$Lambda$4894/0x0000000801b37a40.get(Unknown Source)
at app//org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:73)
at app//org.opensearch.action.ActionRunnable$$Lambda$3917/0x0000000801941850.accept(Unknown Source)
at app//org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88)
at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at app//org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at app//org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at java.base@17.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base@17.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base@17.0.4/java.lang.Thread.run(Thread.java:833)
Locked ownable synchronizers:
- locked <609d1ba2> (a java.util.concurrent.ThreadPoolExecutor$Worker)

     

    opensearch[pgsca2x1064.nfra.io][http_server_worker][T#23]


thread Id : opensearch[pgsca2x1064.nfra.io][http_server_worker][T#23]

state : RUNNABLE


stackTrace:
java.lang.Thread.State: RUNNABLE
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:343)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:237)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:159)
at org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.getResolvedIndexPattern(ConfigModelV7.java:778)
at org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.attemptResolveIndexNames(ConfigModelV7.java:763)
at org.opensearch.security.securityconf.ConfigModelV7.lambda$impliesTypePerm$4(ConfigModelV7.java:1011)
at org.opensearch.security.securityconf.ConfigModelV7$$Lambda$5647/0x0000000801c0ef08.apply(Unknown Source)
at java.base@17.0.4/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base@17.0.4/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
at java.base@17.0.4/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base@17.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base@17.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base@17.0.4/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base@17.0.4/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at org.opensearch.security.securityconf.ConfigModelV7.impliesTypePerm(ConfigModelV7.java:1012)
at org.opensearch.security.securityconf.ConfigModelV7$SecurityRoles.impliesTypePermGlobal(ConfigModelV7.java:489)
at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:460)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:303)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
at app//org.opensearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:632)
at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at app//org.opensearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:774)
at app//org.opensearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:309)
at app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:220)
at app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:118)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:317)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
at app//org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
at app//org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
at app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:426)
at app//org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:502)
at app//org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111)
at app//org.opensearch.rest.action.document.RestBulkAction$$Lambda$5593/0x0000000801c12170.accept(Unknown Source)
at app//org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125)
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:127)
at app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:312)
at app//org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398)
at app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:241)
at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
at app//org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366)
at app//org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445)
at app//org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356)
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55)
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base@17.0.4/java.lang.Thread.run(Thread.java:833)
Locked ownable synchronizers:
- None

     

    opensearch[pgsca2x1064.nfra.io][http_server_worker][T#25]


thread Id : opensearch[pgsca2x1064.nfra.io][http_server_worker][T#25]

state : RUNNABLE


stackTrace:
java.lang.Thread.State: RUNNABLE
at java.base@17.0.4/java.util.stream.AbstractPipeline.wrapSink(AbstractPipeline.java:547)
at java.base@17.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base@17.0.4/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
at java.base@17.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base@17.0.4/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647)
at app//org.opensearch.common.logging.HeaderWarning.getXOpaqueId(HeaderWarning.java:333)
at app//org.opensearch.common.logging.DeprecationLogger$DeprecationLoggerBuilder.withDeprecation(DeprecationLogger.java:116)
at app//org.opensearch.common.logging.DeprecationLogger.deprecate(DeprecationLogger.java:105)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.lambda$checkSystemIndexAccess$3(IndexNameExpressionResolver.java:374)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver$$Lambda$3986/0x0000000801957bd0.accept(Unknown Source)
at java.base@17.0.4/java.util.ArrayList.forEach(ArrayList.java:1511)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.checkSystemIndexAccess(IndexNameExpressionResolver.java:373)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:360)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:237)
at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:159)
at org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.getResolvedIndexPattern(ConfigModelV7.java:778)
at org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.attemptResolveIndexNames(ConfigModelV7.java:763)
at org.opensearch.security.securityconf.ConfigModelV7.lambda$impliesTypePerm$4(ConfigModelV7.java:1011)
at org.opensearch.security.securityconf.ConfigModelV7$$Lambda$5647/0x0000000801c0ef08.apply(Unknown Source)
at java.base@17.0.4/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base@17.0.4/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
at java.base@17.0.4/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base@17.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base@17.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base@17.0.4/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base@17.0.4/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at org.opensearch.security.securityconf.ConfigModelV7.impliesTypePerm(ConfigModelV7.java:1012)
at org.opensearch.security.securityconf.ConfigModelV7$SecurityRoles.impliesTypePermGlobal(ConfigModelV7.java:489)
at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:460)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:303)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
at app//org.opensearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:632)
at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at app//org.opensearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:774)
at app//org.opensearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:309)
at app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:220)
at app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:118)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:317)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78)
at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
at app//org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
at app//org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
at app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:426)
at app//org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:502)
at app//org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111)
at app//org.opensearch.rest.action.document.RestBulkAction$$Lambda$5593/0x0000000801c12170.accept(Unknown Source)
at app//org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125)
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:127)
at app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:312)
at app//org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398)
at app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:241)
at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
at app//org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366)
at app//org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445)
at app//org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356)
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55)
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base@17.0.4/java.lang.Thread.run(Thread.java:833)
Locked ownable synchronizers:
- None

     

    It seems high CPU usage could be contributing to the SSL handshake delays. You may want to check the server’s CPU utilization to confirm this. If CPU usage is indeed high, try optimizing your OpenSearch transport settings. 

    Thanks.

    Got something else on mind? Post Your Question

    Edit your question

    Separate with comma, space or ENTER key. Special characters are not allowed in tag.

    Edit your Answer

    Edit your Comment