What happened on December 13, 2024?
I collect root causes of date / time bugs and I just solved one that was driving me crazy!
Background
I was looking through Wikipedia’s list of time formatting and storage bugs when I noticed an entry for December 2024:
In December 2024, a 30-year-old bug was found in all versions of HCL Notes. When the server is started on or after December 13, 2024, an overflow would prevent the mail router from loading its configuration, and so no mail is delivered. Patches were released on the next day for all supported versions.
With a reference that goes here:
The page mentions:
The date/time issue made the TimeDateDifference() internal call return incorrect results under certain conditions, and dates back to the core Notes program in 1986.
But no actual root cause analysis is provided.
A common source of date / time bugs is overflow, where the date or time eventually grows too large to be contained by its storage, and is then treated as negative or as an error.
I wonder if what happened on December 13, 2024 was an overflow problem or something else?
Interesting. I have been nerd-sniped.
Initial research
I do not have a copy of Notes or Domino to reverse-engineer and it is unlikely that I will get copies. I must work through public sources of information.
What is the TimeDateDifference function that is mentioned in the alert?
Here is its documentation:
#include <misc.h>
LONG LNPUBLIC TimeDateDifference(
const TIMEDATE far *t1,
const TIMEDATE far *t2);
And documentation for the TIMEDATE struct is here:
typedef struct tagTIMEDATE {
DWORD Innards[2];
} TIMEDATE;
Hmmm, no real documentation for the structure of TIMEDATE.
More research
There are other date / time related support pages from HCL:
Error “Unable to interpret time or date” when entering specific dates from the year 1752
There is special handling for invalid dates in the Gregorian Calendar.
Maybe the bug is related to the year 1752 and Gregorian calendar?
The number of seconds from September 1752 to Dec 13 2024 is roughly 2^33, but the days don’t line up with anything significant.
Some Wolfram Language code:
In[1]:= FromUnixTime[UnixTime[{2024, 12, 13}] - 2^33]
Out[1]= Fri 29 Sep 1752 11:03:28GMT-4
(The functions UnixTime and FromUnixTime are a convenient way to convert between dates and seconds.)
The coincidence of September 1752 and 2^33 seconds is tantalizingly close, but not close enough for my liking.
2^33 is not likely to be significant to date / time code from the 1980s and September 29 1752 has no actual significance.
Maybe related to Michaelmas?
In the mean time
I left a comment on HCL’s Community forum for the December 13 bug:
- https://support.hcl-software.com/community?id=community_blog&sys_id=4a7f878047a69e5033723c5c716d4379
But did not receive a reply from HCL.
I also sent an email to Origina but did not receive a reply.
More research
A breakthrough!
Daniel Nashed’s excellent blog explained several things:
TIMEDATE uses Julian Dates.
OK, some things are making sense.
Julian Dates start counting from January 1, 4713 BC (Julian Calendar).
We do not know the exact time that the problem occurred, so let’s look at the Julian Dates for December 13 and December 14 and see if anything interesting happens between them.
In[2]:= Block[{$TimeZone = 0},
Floor[JulianDate[{2024, 12, 13, 12, 0, 0}]]
]
Out[2]= 2460658
In[3]:= Block[{$TimeZone = 0},
Floor[JulianDate[{2024, 12, 14, 12, 0, 0}]]
]
Out[3]= 2460659
The Julian Date for December 13, 2024 is 2460658.
The Julian Date for December 14, 2024 is 2460659.
The number of seconds since the Julian Date epoch to December 13, 2024:
In[4]:= december13JulianSeconds = 2460658 * 24 * 60 * 60
Out[4]= 212600851200
The number of seconds since the Julian Date epoch to December 14, 2024:
In[5]:= december14JulianSeconds = 2460659 * 24 * 60 * 60
Out[5]= 212600937600
What are these values in binary?
In[6]:= IntegerString[december13JulianSeconds, 2]
Out[6]= 11000101111111111111111000101100000000
In[7]:= IntegerString[december14JulianSeconds, 2]
Out[7]= 11000110000000000000001101110010000000
Wait. It looks like a lot of bits get flipped when going from December 13 to December 14.
These values are 38 bits. What are the values when truncated to 32 bits?
In[8]:= IntegerString[212600851200, 2, 32]
Out[8]= 01111111111111111000101100000000
In[9]:= IntegerString[212600937600, 2, 32]
Out[9]= 10000000000000001101110010000000
Ah ha!
The MSB is flipping from 0 to 1 when counting seconds from December 13 to December 14.
Everything makes sense now.
My current hypothesis:
On December 13, 2024 at 8:19:12am (29952 seconds after midnight), the number of seconds since the Julian Date epoch flipped from being positive to being negative in signed 32-bit storage.
Accidentally treating time as negative is a classic cause of date / time bugs.
I hesitate to say “overflow” because it first overflowed January 19, 4645 BC (Julian calendar) and after that, every ~68.1 years, has been alternating between being stored as positive and being stored as negative.
In[8]:= 2^31 / 60 / 60 / 24 / 365 // N
Out[8]= 68.0963
We just happened to be in a window of ~68.1 years where the time was stored as positive.
The previous flip was November 25, 1956, where dates flipped from negative to positive.
This flip was December 13, 2024, where dates flipped from positive to negative.
The next flip will be December 31, 2092, where dates will flip from negative to positive again.
This is the same problem as the upcoming Year 2038 problem, just shifted.
Counting the number of seconds from January 1, 1970 will get to January 19, 2038 before flipping from positive to negative in signed 32-bit storage.
January 19, 2038 will be the first flip since the Unix epoch. But December 13, 2024 was the 99th flip since the Julian Date epoch!
FYI, the FromUnixTime and UnixTime functions from above use arbitrary-precision numbers and do not run into the 32-bit limitations discussed here.
As long as the arguments to TimeDateDifference are in the same positive window (e.g., between November 25, 1956 (after the flip) and December 13, 2024 (before the flip)), then TimeDateDifference probably works fine. This is probably what the critical alert meant when it said:
The date/time issue made the TimeDateDifference() internal call return incorrect results under certain conditions, and dates back to the core Notes program in 1986.
But I am surprised that it took until the actual flipping date to notice the problem. Apparently no dates before and after December 13, 2024 were ever differenced. Or maybe they were and nobody noticed?
Summary
With minimal information, we were able to figure out what happened on December 13, 2024 with high likelihood.
There was a massive red herring to do with September 1752 and the switch to the Gregorian Calendar. But that is part of the fun of figuring these things out.
Ultimately, the number of seconds from the Julian Date epoch flipping signs in signed 32-bit storage every ~68.1 years explains the problem satisfactorily.