Skip to content

Add timestamps to management, server and agent .err logs #12967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dheeraj12347 wants to merge 3 commits into
base: main
Choose a base branch
from
+108 −25
Open

Add timestamps to management, server and agent .err logs #12967

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
45c3bbe
Add timestamps to management, server and agent .err logs
dheeraj12347 Apr 6, 2026
36060f0
Add timestamps to stderr console logs for agent, management and API s…
dheeraj12347 Apr 14, 2026
9315f98
usage: add timestamped stderr console logging
dheeraj12347 Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions agent/conf/log4j-cloud.xml.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at


http://www.apache.org/licenses/LICENSE-2.0


Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
Comment on lines 8 to 16
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
Expand All @@ -20,10 +22,12 @@ under the License.
<Configuration monitorInterval="60">
<Appenders>


<!-- ================================= -->
<!-- Preserve messages in a local file -->
<!-- ================================= -->


<!-- A time/date based rolling appender -->
<RollingFile name="FILE" append="true" fileName="@AGENTLOG@" filePattern="@AGENTLOG@.%d{yyyy-MM-dd}.gz">
<ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
Expand All @@ -33,44 +37,52 @@ under the License.
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{3}] (%t:%x) (logid:%X{logcontextid}) %m%ex%n"/>
</RollingFile>


<!-- ============================== -->
<!-- Append messages to the console -->
<!-- ============================== -->


<Console name="CONSOLE" target="SYSTEM_OUT">
<ThresholdFilter level="OFF" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%-5p [%c{3}] (%t:%x) (logid:%X{logcontextid}) %m%ex%n"/>
<ThresholdFilter level="ERROR" onMatch="DENY" onMismatch="ACCEPT"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{3}] (%t:%x) (logid:%X{logcontextid}) %m%ex%n"/>
</Console>

<Console name="CONSOLE_ERR" target="SYSTEM_ERR">
<ThresholdFilter level="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{3}] (%t:%x) (logid:%X{logcontextid}) %m%ex%n"/>
</Console>
Comment on lines 46 to 54
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same concern here: if .err is sourced from stderr as stated in the PR description, this console appender targeting SYSTEM_OUT won’t influence .err formatting. Consider using target=\"SYSTEM_ERR\" (or aligning systemd routing) to ensure .err lines include the new timestamp.

Copilot uses AI. Check for mistakes.
Comment on lines 46 to 54
</Appenders>


<Loggers>


<!-- ================ -->
<!-- Limit categories -->
<!-- ================ -->

<Logger name="com.cloud" level="INFO"/>

<Logger name="com.cloud" level="INFO"/>
<Logger name="org.apache" level="INFO"/>

<Logger name="org" level="INFO"/>

<Logger name="net" level="INFO"/>

<Logger name="com.amazonaws" level="INFO"/>

<Logger name="httpclient.wire" level="INFO"/>

<Logger name="org.apache.http.wire" level="INFO"/>


<!-- ======================= -->
<!-- Setup the Root category -->
<!-- ======================= -->


<Root level="INFO">
<AppenderRef ref="CONSOLE"/>
<AppenderRef ref="CONSOLE_ERR"/>
<AppenderRef ref="FILE"/>
</Root>


</Loggers>
</Configuration>
42 changes: 40 additions & 2 deletions client/conf/log4j-cloud.xml.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at


http://www.apache.org/licenses/LICENSE-2.0


Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
Comment on lines 8 to 16
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
Expand All @@ -20,14 +22,17 @@ under the License.
<Configuration monitorInterval="60" packages="org.apache.cloudstack.alert.snmp,org.apache.cloudstack.syslog">
<Appenders>


<properties>
<property name="filters">net.sf.cglib.proxy</property>
</properties>


<!-- ================================= -->
<!-- Preserve messages in a local file -->
<!-- ================================= -->


<!-- A regular appender -->
<RollingFile name="FILE" append="true" fileName="@MSLOG@" filePattern="@MSLOG@.%d{yyyy-MM-dd}.gz">
<ThresholdFilter level="TRACE" onMatch="ACCEPT" onMismatch="DENY"/>
Expand All @@ -46,89 +51,122 @@ under the License.
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) (logid:%X{logcontextid}) %m%ex{filters(${filters})}%n"/>
</RollingFile>


<!-- ============================== -->
<!-- Append warnings+ to the syslog if it is listening on UDP port -->
<!-- ============================== -->


<Syslog name="SYSLOG" host="localhost" facility="LOCAL6">
<ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) (logid:%X{logcontextid}) %m%ex{filters(${filters})}%n"/>
</Syslog>


<!-- ============================== -->
<!-- Append alerts to the syslog if it is configured -->
<!-- ============================== -->


<AlertSyslogAppender name="ALERTSYSLOG" syslogHosts="" facility="LOCAL6">
<ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) (logid:%X{logcontextid}) %m%ex{filters(${filters})}%n"/>
</AlertSyslogAppender>


<!-- ============================== -->
<!-- Append messages to the console -->
<!-- ============================== -->


<Console name="CONSOLE" target="SYSTEM_OUT">
<ThresholdFilter level="OFF" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%-5p [%c{1.}] (%t:%x) (logid:%X{logcontextid}) %m%ex{filters(${filters})}%n"/>
<ThresholdFilter level="ERROR" onMatch="DENY" onMismatch="ACCEPT"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) (logid:%X{logcontextid}) %m%ex{filters(${filters})}%n"/>
</Console>
Comment on lines 82 to 85
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as server: the description references stderr->.err, but the console appender is configured for SYSTEM_OUT. If .err is populated from stderr, the timestamped pattern won’t be reflected there unless the target is SYSTEM_ERR (or the unit routes stdout to .err).

Copilot uses AI. Check for mistakes.

<Console name="CONSOLE_ERR" target="SYSTEM_ERR">
<ThresholdFilter level="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) (logid:%X{logcontextid}) %m%ex{filters(${filters})}%n"/>
</Console>
Comment on lines 82 to +90


<!-- ============================== -->
<!-- send alert warnings+ as the SNMP trap if it is configured! -->
<!-- ============================== -->


<SnmpTrapAppender name="SNMP" SnmpManagerIpAddresses="" SnmpManagerPorts="" SnmpManagerCommunities="">
<ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY"/>
</SnmpTrapAppender>
</Appenders>


<Loggers>


<Logger name="com.cloud" level="DEBUG"/>


<Logger name="org.apache.cloudstack" level="DEBUG"/>


<Logger name="com.cloud.utils.nio" level="INFO"/>


<Logger name="org.apache" level="INFO"/>


<Logger name="org.apache.cloudstack.api.command" level="DEBUG"/>


<Logger name="org" level="INFO"/>


<Logger name="org.springframework" level="WARN"/>


<Logger name="org.apache.cloudstack.spring.module.context.ResourceApplicationContext" level="WARN"/>


<Logger name="net" level="INFO"/>


<Logger name="apiserver.com.cloud" level="DEBUG"/>


<Logger name="apiserver.com.cloud" level="DEBUG" additivity="false">
<AppenderRef ref="APISERVER"/>
</Logger>


<Logger name="com.amazonaws" level="INFO"/>


<Logger name="httpclient.wire" level="INFO"/>


<Logger name="org.apache.cloudstack.alert" additivity="false" level="WARN">
<AppenderRef ref="SYSLOG"/>
<AppenderRef ref="CONSOLE"/>
<AppenderRef ref="CONSOLE_ERR"/>
<AppenderRef ref="FILE"/>
<AppenderRef ref="SNMP"/>
<AppenderRef ref="ALERTSYSLOG"/>
</Logger>


<!-- ======================= -->
<!-- Setup the Root category -->
<!-- ======================= -->


<Root level="INFO">
<AppenderRef ref="SYSLOG"/>
<AppenderRef ref="CONSOLE"/>
<AppenderRef ref="CONSOLE_ERR"/>
<AppenderRef ref="FILE"/>
</Root>


</Loggers>
</Configuration>
32 changes: 30 additions & 2 deletions server/conf/log4j-cloud.xml.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at


http://www.apache.org/licenses/LICENSE-2.0


Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
Comment on lines 8 to 16
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
Expand All @@ -20,6 +22,7 @@ under the License.
<Configuration monitorInterval="60">
<Appenders>


<!-- ================================= -->
<!-- Preserve messages in a local file -->
<!-- ================================= -->
Expand Down Expand Up @@ -52,6 +55,7 @@ under the License.
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) %m%ex%n"/>
</RollingFile>


<!-- ============================== -->
<!-- Append warnings+ to the syslog if it is listening on UDP port -->
<!-- ============================== -->
Expand All @@ -62,60 +66,84 @@ under the License.
<PatternLayout pattern="%-5p [%c{1.}] (%t:%x) %m%ex%n"/>
</Syslog>


<!-- ============================== -->
<!-- Append messages to the console -->
<!-- ============================== -->


<Console name="CONSOLE" target="SYSTEM_OUT">
<ThresholdFilter level="OFF" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%-5p [%c{1.}] (%t:%x) %m%ex%n"/>
<ThresholdFilter level="ERROR" onMatch="DENY" onMismatch="ACCEPT"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) %m%ex%n"/>
</Console>

<Console name="CONSOLE_ERR" target="SYSTEM_ERR">
<ThresholdFilter level="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d{DEFAULT} %-5p [%c{1.}] (%t:%x) %m%ex%n"/>
Comment on lines 75 to +82
Copy link
Copy Markdown
Contributor

@DaanHoogland DaanHoogland Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to double check this, @dheeraj12347 . I looked in an env and the .out file already contains the date stamps for each line. The .err file doesn’t .

Copy link
Copy Markdown
Contributor Author

@dheeraj12347 dheeraj12347 Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to double check this, @dheeraj12347 . I looked in an env and the .out file already contains the date stamps for each line. The .err file doesn’t .

You’re right, Daan – .out already had timestamps while .err didn’t. I’ve now added a CONSOLE_ERR appender using SYSTEM_ERR with the same timestamped pattern, so stderr (and thus the .err files) will get timestamps as well.

Comment on lines +79 to +82
Copy link
Copy Markdown
Contributor Author

@dheeraj12347 dheeraj12347 Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve now updated usage/conf/log4j-cloud_usage.xml.in to add a CONSOLE_ERR appender targeting SYSTEM_ERR with a timestamped pattern, so usage.err will also get timestamps in line with issue #12887.

Comment on lines 75 to +82
</Console>
Comment on lines 75 to 83
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says the .err logs are fed from console/stderr, but this Console appender targets SYSTEM_OUT. If systemd is capturing stderr into .err, changing the stdout pattern won’t affect .err. Consider switching the console target to SYSTEM_ERR (or update the systemd unit routing) so the timestamped console output actually lands in .err.

Copilot uses AI. Check for mistakes.
</Appenders>


<Loggers>


<Logger name="com.cloud" level="DEBUG"/>


<Logger name="org.apache.cloudstack" level="DEBUG"/>


<Logger name="org.apache" level="INFO"/>


<Logger name="org" level="INFO"/>


<Logger name="net" level="INFO"/>


<Logger name="apiserver.com.cloud" level="DEBUG"/>


<Logger name="apiserver.com.cloud" level="DEBUG" additivity="false">
<AppenderRef ref="APISERVER"/>
</Logger>


<Logger name="com.amazonaws" level="INFO"/>


<Logger name="httpclient.wire" level="INFO"/>


<!-- ============================== -->
<!-- Add or remove these logger for SNMP, this logger is for SNMP alerts plugin -->
<!-- ============================== -->


<Logger name="org.apache.cloudstack.alerts" additivity="false" level="WARN">
<AppenderRef ref="SYSLOG"/>
<AppenderRef ref="CONSOLE"/>
<AppenderRef ref="CONSOLE_ERR"/>
<AppenderRef ref="FILE"/>
<AppenderRef ref="SNMP"/>
<AppenderRef ref="ALERTSYSLOG"/>
</Logger>


<!-- ======================= -->
<!-- Setup the Root category -->
<!-- ======================= -->


<Root level="INFO">
<AppenderRef ref="SYSLOG"/>
<AppenderRef ref="CONSOLE"/>
<AppenderRef ref="CONSOLE_ERR"/>
<AppenderRef ref="FILE"/>
<AppenderRef ref="INFO-FILE"/>
</Root>


</Loggers>
</Configuration>
Loading
Loading