How to ensure that allocation failure doesn't happen in Young generation without a...

I don't want to limit the size of metaspace. However I want to make use of as much as possible from the available memory in the operating system. 

It seems that OS has less free memory that is why it is not able to us as much as possible. Correct me if I am wrong ?

At the moment, I decrease the heap size so that more free memory is available in the system to be used by Metaspace, and this really seems to improve the performance by decreasing the GC pause time.

My question is if I don't set the metaspace size and if it start using it all of the possible memory then what are the implication of this situation.

How this memory space is freed up if further need is raised by the application ?

>At the moment, I decrease the heap size so that more free 
>memory is available in the system to be used by Metaspace, and 
>this really seems to improve the performance by decreasing the 
>GC pause time.

 It's very possible. But here is the relationship you might have to understand: If the host/container in which your application is running is crowded with several processes and they all compete for memory then 'Metadata GC Threshold' GC pause times will go higher. Since you are not setting '-XX:MetaspaceSize' argument, default value '20mb' (in your platform) will be used. 20mb is too small number for modern applications. As your JVM starts to load new classes, your JVM's metaspace memory requirement will also start to grow. Inorder to allocate memory to metaspace region, JVM will start to request additional memory from the Operating system. Whenever metaspace region size has to expand it will trigger full GC (i.e. stop the world GC) pause. If OS is running low on memory then GC pause times will become higher. Because OS has to swap less frequently processes to disk , compact fragmented memory, and make room for your request. All these take time. Thus it will ultimately increase GC pause time incurred by your application. 
 
 When you reduced your application's heap size, demand for memory in OS has gone down and ultimately 'Metadata GC Threshold' GC events pause time has also got reduced. But note reducing JVM heap size has to be cautiously done, as it can impact other GC/performance characteristics of the application.
 
>It seems that OS has less free memory that is why it is not 
>able to us as much as possible. Correct me if I am wrong ?

 If you can run yCrash on your production server, it will show us overall memory consumption, swap memory consumption, each individual processes memory consumption. It might help us to eliminate unnecessary processes or allocate less memory for other processes. You can register here to get a free 14 day trial of yCrash.
  
>My question is if I don't set the metaspace size and if it start 
>using it all of the possible memory then what are the implication 
>of this situation.

 I would vote against not setting '-XX:MaxMetaspaceSize' and '-XX:MetaspaceSize' in enterprise grade applications, for following reasons:
 
a. Increase Garbage Collection pause time

If you don't pass '-XX:MetaspaceSize' JVM argument then default value will be used. Default value is platform dependant and varies from 12mb to 20mb. 20mb is *very small number* for modern applications. As your JVM starts to load new classes, JVM's metaspace memory requirement will also start to grow. Inorder to allocate memory to metaspace region, JVM will start to request additional memory from the Operating system. Whenever metaspace region size has to expand it will trigger full GC (i.e. stop the world GC) pause. If OS is running low on memory then GC pause times will become higher. Because OS has to swap less frequently processes to disk , compact fragmented memory, and make room for your request. All these take time. Thus it will ultimately increase GC pause time incurred by your application. 

b. java.lang.OutOfMemoryError:Metaspace

If you don't pass '-XX:MaxMetaspaceSize' JVM argument then default value will be used. Default value is 'unlimited'. Having unlimited maximum metaspace might sound good at surface level. But consider this scenario: Let's say your application's peak metaspace consumption is 0.5 GB memory and all other memory regions in JVM consume 2GB memory. Let's say your application is running on host(or contianer) which has 4GB of RAM.  Things all might sound good today. 

Let's say tomorrow your metaspace consumption is currently at 0.25GB (it has reached it's peak usage), lets say some new processes got launched on your host/container or existing processes starts to consume more memory and they occupy 1.75 GB of memory. That means overall memory consumption on this host/container will be 4GB (i.e. JVM regions: 2GB + Other processes: 1.75GB + Metapsace: 0.25GB). Memory consumption of this host/container will be at peak. Let's say at this point your application's metaspace consumption starts to grow then it will result in java.lang.OutOfMemoryError:Metaspace. Because your host/container wouldn't be able to allocate additional memory, because it is already at it's peak 4GB utilization. Thus your application will get java.lang.OutOfMemoryError:Metaspace unexpectedly eventhough no code changes were made in your application. 

c. Challenges with capacity planning

If you don't set upper bound on your metaspace size, then how you are going to plan for capacity? How you are going to decide whether you want to go for m3.large EC2 instance or m3.xlarge EC2 instance. Capacity planning would be difficult.