Logging with Linux VPC server instances

Use the IBM® Log Analysis service to monitor and manage logs from a Linux VPC server instance in a centralized logging system on the IBM Cloud.

As of 28 March 2024 the IBM Log Analysis and IBM Cloud Activity Tracker services are deprecated and will no longer be supported as of 30 March 2025. Customers will need to migrate to IBM Cloud Logs, which replaces these two services, prior to 30 March 2025. For information about IBM Cloud Logs, see the IBM Cloud Logs documentation.

These instructions are for Ubuntu Linux systems but can be used for other Linux systems.

You can collect and monitor system and application logs.

By default, the logging agent for Ubuntu monitors log files in the /var/log directory. For example, the Ubuntu system log (/var/log/syslog) is monitored by default.

On the IBM Cloud, configure an Ubuntu server to forward logs to an IBM Log Analysis instance by completing the following steps:

Provision a VPC running Ubuntu Linux.
Provision an instance of the IBM Log Analysis service.
Configure the logging agent in the Ubuntu server.
Optionally, add additional directories to be monitored by the agent.

In this tutorial, you will learn how to configure an Ubuntu server to forward logs to an IBM Log Analysis instance.

Before you begin

Read about IBM Log Analysis. For more information, see About.

Work in a supported region.

You can send data from an Ubuntu instance that is located in the same region as your logging instance, in a different region, or not in the IBM Cloud.

Use a user ID that is a member, or an owner of, an IBM Cloud account. To get an IBM Cloud IBMID, go to: Create an account.

Your IBMID must have assigned IAM policies for each of the following resources in the region that your IBM Log Analysis instance is in:

Your IBMID must have assigned IAM policies for each of the following resources:

List of IAM policies required to complete the tutorial
Resource	Scope of the access policy	Role	Region	Information
Resource group default	Resource group	Viewer	us-south	This policy is required to allow the user to see service instances in the Default resource group.
IBM Log Analysis service	Resource group	Editor	us-south	This policy is required to allow the user to provision and administer the IBM Log Analysis service in the default resource group.

The IBM Cloud CLI must be installed. For more information, see Installing the IBM Cloud CLI.

Provision an Ubuntu Linux VPC server instance

If you have an existing Ubuntu Linux virtual server instance you want to monitor, you can skip this step.

If you don't have a virtual private cloud, use the IBM Cloud console to create VPC resources.
If you don't have an Ubuntu Linux virtual server instance, create an Ubuntu Linux virtual server instance by using the UI and selecting Ubuntu Linux as the Operating System.

Provision an IBM Log Analysis instance

To provision an instance of IBM Log Analysis through the IBM Cloud UI, complete the following steps:

Log in to your IBM Cloud account.

Click Log in to IBM Cloud to sign in to the IBM Cloud.

After you log in with your user ID and password, the IBM Cloud console opens.
Click Catalog. The list of the services that are available in IBM Cloud opens.
Select the Logging and Monitoring category.
Click the IBM Log Analysis tile.
Select a region for the service instance.
Select the Lite service plan.

By default, the Lite plan is set.

For more information about other service plans, see Pricing plans.
Specify a Service name for your IBM Log Analysis service instance.
Select the Default resource group.

By default, the Default resource group is set.
To provision the IBM Log Analysis service in the IBM Cloud selected resource group, click Create.

After you provision an instance, the IBM Log Analysis dashboard opens.

To provision an instance of logging through the CLI, see Provisioning logging through the IBM Cloud CLI.

Configure your Ubuntu server to send logs to your instance

To configure your Ubuntu server to send logs to your IBM Log Analysis instance, you must install a logging-agent. The logging agent reads log files from /var/log or other directories you specify, and forwards the log data to your logging instance.

To configure your Ubuntu server to forward logs to your logging instance, complete the following steps from an Ubuntu terminal:

Install the logging agent. Run the following commands:

echo "deb https://assets.logdna.com stable main" | sudo tee /etc/apt/sources.list.d/logdna.list

wget -O- https://assets.logdna.com/logdna.gpg | sudo apt-key add -

sudo apt-get update

sudo apt-get install logging-agent < "/dev/null"

Set the ingestion key that the logging agent will use to forward logs to the IBM Log Analysis instance.
```
sudo logging-agent -k <INGESTION_KEY>
```
where <INGESTION_KEY> contains the ingestion key for the IBM Log Analysis instance where the logs will be forwarded.

You can retrieve the ingestion key using the IBM Cloud console, or by the IBM Cloud CLI.
Set the authentication endpoint. The logging agent uses this host to authenticate and get the token to forward logs.
```
sudo logging-agent -s LOGDNA_APIHOST=api.us-south.logging.cloud.ibm.com
```

Set the ingestion endpoint.

sudo logging-agent -s LOGDNA_LOGHOST=logs.us-south.logging.cloud.ibm.com

(Optional) Define any additional log paths to be monitored. Run the following command:
```
sudo logging-agent -d <PATH_TO_LOG_FOLDERS>
```
Where <PATH_TO_LOG_FOLDERS> is the path where logs are saved on your system. By default, /var/log is monitored.
(Optional) Configure the logging agent to tag your hosts. Run the following command:
```
sudo logging-agent -t TAG1,TAG2
```
Tags must be separated by commas and without any spaces between the comma and tag name.
Update the logging agent with your changes. Run the following command:
```
sudo update-rc.d logging-agent defaults
```
Start the logging agent. Run the following command:
```
sudo /etc/init.d/logging-agent start
```

Troubleshooting

You can use the /var/log/logdna-agent.log log to determine if there are any issues with your logdna-agent installation.

Launch the logging Web UI

To launch the IBM Log Analysis dashboard from the IBM Cloud UI, complete the following steps:

Log in to your IBM Cloud account.

Click IBM Cloud dashboard to launch the IBM Cloud dashboard.

After you log in with your user ID and password, the IBM Cloud Dashboard opens.
In the navigation menu, select Observability.
Click Logging.

The list of IBM Log Analysis instances that are available on IBM Cloud is displayed.
Select one instance. Then, click Open dashboard.

The logging Web UI opens and displays your cluster logs.

View your logs

From the logging Web UI, you can view your logs as they pass through the system. You view logs by using log tailing.

With the Free service plan, you can only tail your latest logs.

For more information, see Viewing logs.

Next steps

The following additional features are available:

To use any of these features, you must upgrade the IBM Log Analysis plan to a paid plan.