The Problem Report
A programmer I support sent me the following last October:
Hey Jim,
I just found a difference between our windows and unix boxes regarding time that is standing in the way of me finishing […] I’ve been working on and wanted your help.
[Description of test code and environment]
This is Test.java taking a milliseconds since the epoch time: 1256975836665 and turning that into a date. It should give 2009-10-31 00:57:16 but is off by one hour and does if you run the same from a windows box on our network.
I don’t know how to do the same from the command line to see if it’s java specific or machine specific, but was wondering if you could look at those machines and see if perhaps timezone or daylight saving is set incorrectly?
The Investigation
Honestly, it had been a long time since I looked at the time zone configuration on a Linux box. I had helped build out CNET‘s Kickstart infrastructure in 2002 and hadn’t really looked at system settings like time zone since. Upgrading the zoneinfo database to deal with changes was just an RPM upgrade and never required much thought. So, I had to refresh my knowledge. Further, all of the systems I support for 42Lines and our clients I inherited in mid-January, so I didn’t know how they were set up.
The key file here is /etc/localtime. What I found surprising was that instead of a symlink to the correct file in /usr/share/zoneinfo/, it was just a file. This is a problem as the zoneinfo data for a zone is a compiled binary format and not human readable. So, how to determine what zone it was? In cases like this I turn to my trusty friend md5sum.
# md5sum /etc/localtime f3e91959e492f62136812f8f556713a3 /etc/localtime # find /usr/share/zoneinfo/America -type f | xargs md5sum | grep f3e91959e492f62136812f8f556713a3 f3e91959e492f62136812f8f556713a3 /usr/share/zoneinfo/America/Ensenada f3e91959e492f62136812f8f556713a3 /usr/share/zoneinfo/America/Tijuana
Okay, so the timezone is set to Tijuana. Tijuana is in the Pacific Timezone just like San Francisco, right? Sometimes.
I had to coordinate a massive update at CNET in 2007 thanks to Congress changing the DST rules for the United States. So I suspected that that the problem might be due to the new rules starting in 2007. I wasn’t working with these systems during the last DST transition, so it is possible that they just had an old zoneinfo database that didn’t include the recent rule changes.
I checked the version of tzdata (rpm -qf /usr/share/zoneinfo/America/Tijuana) and the version post-dated that 2007 changes (tzdata-2009u-1). So, perhaps Mexico didn’t implement the same changes to DST that the United States and Canada did? Again, the zoneinfo database is binary and somewhat to my surprise the RPMs do not install the source. So, I hunted down the source and took a look.
It turns out the Mexico has had a pretty turbulent relationship with DST, far more so than the US. The comments in the zoneinfo source file reflect not only the difficulty in following such things, but the dedication of a few select people to try and translate political intent and law into something logical (a rough task regardless of country.) Here’s what Mexico was using starting in 2002:
Rule Mexico 2002 max - Apr Sun>=1 2:00 1:00 D Rule Mexico 2002 max - Oct lastSun 2:00 0 S
So as soon as the US changed in 2007 to:
Rule US 2007 max - Mar Sun>=8 2:00 1:00 D Rule US 2007 max - Nov Sun>=1 2:00 0 S
Anyone in the United States using Tijuana as their zone had the wrong time during DST. It turns out that the border cities in Mexico will change to US rules in 2010.
My response to the programmer:
This is because the TZ on that machine is set to America/Tijuana instead of US/Pacific and the US (but not Mexico) changed the DST rules starting for 2007. 10/30 falls in the window between the old DST ending date (last Sunday in Oct) and the current ending date (first Sun in Nov) for the US. Thanks, Congress.
Another programmer chimed in with the witty comment:
Those machines are not allowed to be in Tijuana according to our health care plan. If their drives crash, they won’t be covered.
The Solution
Ensure your time zone is set correctly. We changed to US/Pacific. I also ensured that /etc/localtime was a symlink to the appropriate zoneinfo file to make things a bit more obvious.