The Cloud Foundry cf push and log commands return an error

The Cloud Foundry cf push and cf log commands fail.

Symptoms

The Cloud Foundry cf push APPLICATION_NAME and cf log APPLICATION_NAME --recent return errors when they try to reach the Cloud Foundry Loggregator.

Running cf push APPLICATION_NAME displays the following error message:

Warning: error tailing logs

Running cf logs APPLICATION_NAME --recent displays the following error message:

unknown issue when making HTTP request to Loggregator

Resolving the problem

1. Log in to BOSH. See Command line interfaces for IBM® Cloud Private Cloud Foundry.

2. Determine which Loggregator instance is failing. For each nats instance in your installation, run the following command. For developer installations, check nats/0, and for enterprise installations, check nats/0, nats/1 and nats/2.

   bosh -e IBMCloudPrivate -d Bluemix ssh <nats> -c "tail /var/vcap/sys/log/loggregator_trafficcontroller/loggregator_trafficcontroller.stderr.log"
   

   Where <nats> is the instance to check.

   The command output for the nats instance that causes the problem contains one of the following error messages:

   2017/12/05 18:37:12 Could not get app information: [Get http://api.local.bluemixx
   .net/internal/log_access/f3c629e8-0bf0-4ecb-98a6-5dd42b707acb: dial tcp: lookup  
   api.local.bluemix.net on 127.0.0.1:53: no such host]
   

   or

   2017/12/05 18:45:04 Error while reading from stream (192.168.248.11:8082): rpc ee
   rror: code = 1 desc = context canceled
   2017/12/05 18:45:04 Unable to connect to doppler (192.168.248.11:8082): rpc erroo
   r: code = 1 desc = context canceled
   2017/12/05 18:45:04 Disconnecting from stream (192.168.248.10:8082) (doppler.diss
   connect=false) (ctx.disconnect=1)
   2017/12/05 18:45:04 Disconnecting from stream (192.168.248.9:8082) (doppler.discc
   onnect=false) (ctx.disconnect=1)
   2017/12/05 18:45:04 Disconnecting from stream (192.168.248.11:8082) (doppler.diss
   connect=false) (ctx.disconnect=1)
   

3. Restart the Loggregator. Run the following commands:

   bosh -e IBMCloudPrivate -d Bluemix ssh <nats>
   sudo su -
   monit restart unbound
   monit summary
   monit restart loggregator_trafficcontroller
   exit
   

   Where <nats> is the nats instance that produced the error.

4. Repeat the previous step for each nats instance that displayed the error message.