Elasticsearch / Indexが作成されない （Too many open files）

elasticsearch

DEBUGレベルでログを出力していると、エラーのなかで「Too many open files」のエラーメッセージを確認することがあります。

[2014-09-28 01:23:45,678][DEBUG][action.bulk              ] [Maverick] [hoge-2014.09.28][3] failed to execute bulk item (index) index {[hoge-2014.09.28][access][k_0ABC3KrXCC7sv006eW9a], source[{"host":"192.168.0.1","user":"-","method":"GET","path":"/","status":"200","size":"3006","referer":"http://example.com/","agent":"Mozilla/5.0 (Windows NT 6.1; rv:28.0) Gecko/20100101 Firefox/28.0","track-cookie":"-","request-time":"4102","x-forwarded-for":"192.168.0.2","@log_name":"hoge.access","@timestamp":"2014-09-28T01:23:45+09:00"}]}
org.elasticsearch.index.engine.CreateFailedEngineException: [hoge-2014.09.28][3] Create failed for [access#k_0ABC3KrXCC7sv006eW9a]
        at org.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:389)
        at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:382)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:153)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: /usr/local/elasticsearch/data/hoge/nodes/0/indices/hoge-2014.09.28/3/index/_9.fdx (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
...

ファイルディスクリプタが上限に達してしまっている可能性が高いので、「ulimit -n」で現在の「open files」上限値を確認し、必要に応じて拡張します(root作業)。

具体的には ulimit コマンドでの変更(即時反映・サーバ再起動までは有効)、必要に応じて /etc/security/limits.conf(サーバ起動時の初期値)の変更を実施します。