Omnitracs' Road Ahead blog

Ongoing updates regarding the recent MCP connectivity issue

headshot of omnitracs ceo ray greer
Ray Greer
Chief Executive Officer
Thumbnail

 

 

 

 

Update: Thursday, November 21, 2019, 8:35 pm CST

Below is the latest update on MCP final fixes.

MCP 50:  The final fix continues to propagate to MCP 50s. However, I no longer believe that ALL on air devices will be upgraded by the end of this week. The process is taking longer than anticipated due to the file size and the fact that we must send all software components to a portion of the devices over the air. We have changed power down settings on all devices to four hours extending the available download windows. As of tonight, 44% of all MCP50s are complete, and another 33% have received more than 75% of the components. The timing to get all devices upgraded remains difficult to project but based on current progress, it looks like it could be up to another week.

However, there are two ways you can expedite the process: 1) devices to spend more non-peak time (nights and weekends) on air, and 2) “stick upgrade” devices with the software via memory stick. This process takes less than 20 minutes per device, and detailed instructions and software can be downloaded by your team on the Omnitracs Customer Portal under Support -> Firmware/OS -> MCP50 firmware. This second option is ideal for dedicated fleets that originate from constant locations.

MCP 110/MCP 200:  The final fix for these devices is now officially in beta. We will follow our standard beta process upgrading a growing subset of devices in the field over a few days with careful review and analysis. Assuming all goes as planned, we’ll move the MCP 200 devices to the deployment phase on Tuesday, Nov 26th and MCP 110 devices on Thursday, Nov 28th.

Please see the updated dashboard above.

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Monday, November 18, 2019, 5:35 pm CST

Here is the most recent update on MCP fixes.

MCP 50:  The final fix continues to propagate to MCP 50s in a measured way. Assuming the current deployment rate holds true, I expect completion for on-air devices by Friday.

MCP 110:  The engineering team has moved to limited field testing for both GOBI 1000 and GOBI 3000 chipsets. I expect we’ll move to full beta on Thursday.

MCP 200:  Same update as for MCP 110s.  I expect full beta for the MCP 200 final fix to also begin on Thursday.

Please see the updated dashboard above.

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Saturday, November 16, 2019, 4:30 pm CST

Here is the latest update on MCP fixes.

MCP 50:  The final fix continues to propagate to MCP 50 devices in a measured way. Based on the current rate, I don’t expect completion across the broad device population until next week.

MCP 110:  Through our monitoring of the MCP 110s with GOBI 3000 chipsets, we identified an edge case that we feel is best addressed prior to expanding the beta. We are working over the weekend to address the issue and expect to return to field testing early next week. Development for the MCP 110s with GOBI 1000 chipsets continues to progress.

MCP 200:  We moved to a limited beta for MCP 200s with the GOBI 3000 chipset on Friday and will continue to monitor. We expect to know more about our ability to expand the beta on Monday. As with the MCP 110s, the devices with the GOBI 1000 chipset will follow shortly after the 3000s.

Please see our updated dashboard above.

 

Respectfully,

Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Thursday, November 14, 2019, 5:45 pm CST

Below is my update for MCP fixes as of today.

MCP 50:  Deployment of the MCP 50 final fix continues to progress, and at the current pace, I don’t expect completion across the entire device population until Monday afternoon.

MCP 110:  We began early field testing to a limited group of MCP 110 devices with GOBI 3000 chipsets today with the GOBI 1000 chipsets to follow soon. We will monitor results and add devices to this test as appropriate.

MCP 200:  I expect to share news about our testing plan for MCP 200s tomorrow.

Please see our updated dashboard above.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Wednesday, November 13, 2019, 5:15 pm CST

Here is today’s MCP update.

MCP 50 devices:  We have initiated deployment of the final fix for all MCP 50 devices and are carefully monitoring the results. Broad completion is expected by end of day on Friday.

MCP 110 and MCP 200 devices:  We remain on schedule to initiate field testing (beta) to a limited subset of MCP 110 and MCP 200 devices in both the US and Canada on Thursday afternoon. I expect the MCP 110/MCP 200 beta to take longer than the MCP 50 beta because of the larger variances across the device population.

Please see the updated dashboard above.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Tuesday, November 12, 2019, 5:40 pm CST

Here is today’s progress report for MCP 50, MCP 110 and MCP 200 devices.

MCP 50 devices:  The MCP 50 beta test for the final fix is complete, was successful, and is now being deployed to all these devices. We expect to have all “on-air” MCP 50s updated by end of day Friday. Devices that come “on-air” later will have the update waiting for them.

MCP 110 and MCP 200 devices:  Additional progress was made in the development phase for the MCP 110s and MCP 200s. Continuing at this pace should allow us to begin field testing in the US and Canada on Thursday.

Please see the updated dashboard above.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Monday, November 11, 2019, 5:31 pm CST

This update is to share today’s progress in the final remediation for all five MCP/GOBI combinations.

For the MCP 50 devices, we expanded beta testing today to 2000 devices. To the extent we continue to see success, we will deploy to all MCP 50s on Tuesday (tomorrow).

For the MCP 110 and MCP 200 devices, additional progress was made in the development phase for each. I expect to be able to share more tangible progress tomorrow.

Please see the updated dashboard above.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Sunday, November 10, 2019, 4:35 pm CST

I’m sharing this update to share today’s progress in the final remediation for all five MCP/GOBI combinations.

For the MCP 50 devices, beta testing began Saturday night and has been successful. We expanded the number of devices in beta today and will continue through Monday – as we analyze the results.

For the MCP 110 and MCP 200 devices, additional progress was made in the development phase.

See the updated dashboard above.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Saturday, November 9, 2019, 4:18 pm CST

Progress continues in the final remediation for all five MCP/GOBI combinations. Lab testing was successful overnight and transitioned to truck testing for the MCP 50 devices today. Additional progress was made in the development phase for MCP 110 and MCP 200 devices. The updated dashboard above lays out our progress.

Steps put in place throughout the week continue to enable Omnitracs customers to use Hours of Service and other functionality on MCP devices.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Friday, November 8, 2019, 6:15 pm CST

Since I began communicating with you Sunday evening, dedicated Omnitracs teams have focused on a number of objectives including returning all MCP devices to normal hours of service (HOS) and records of duty status (RODS) operations, all drivers to electronic logs and all other functionality to normal as well. Simultaneously, we remain focused on developing and implementing a return to GPS-enabled time with GPS-enabled positioning.

Throughout, I’ve reinforced the guiding principles of both speed and certainty. While we all want a permanent fix today, we need to balance speed with long-term stability. As such, I commit to you that a permanent fix will be delivered expeditiously, but only after thorough validation and testing.

Hours of service (HOS) and return to electronic logs:  As communicated earlier in the week, the impacted MCP devices are HOS/RODS compliant, and all drivers should now be on electronic logs. Following the GPS rollover event Saturday night, our most immediate and urgent need was to restore accurate date/time at the device. This is complete and devices are fully capable of recording accurate HOS data including position/location data.

Devices coming online for the first time since the event will need a configuration update provided over-the-air or through a USB memory stick. Stragglers will continue to be addressed.

Real-time device communication and other functionality:  We have also implemented back-end enhancements and pushed configuration updates to improve device communication frequency. Although these changes were not needed for HOS/RODS compliance, they do improve the navigation, workflow, messaging and HOS experience. Customers will continue to see improvements in these areas as trucks receive configuration updates via cellular communication. For trucks that start up outside of cellular coverage, the device’s ability to communicate has now been reduced to 15 minutes. As a result, you should see your devices communicating in real time.

Return to GPS-enabled time with GPS-enabled positioning:  The root cause of this issue is related to firmware code for both the GOBI 1000 and GOBI 3000 communications chipsets. MCPs use these chipsets for communicating with GPS satellites for date/time and positioning information at the device, as well as cellular communication networks. The “GPS rollover issue” impacted the date/time function, not the location/positioning function.

Three models of our MCP devices, utilizing two different communications chipsets, were impacted – the MCP 50, (GOBI 3000), MCP 110 (GOBI 3000 and 1000) and MCP 200 (GOBI 3000 and 1000). This means that fixes must be assessed, designed, developed, built, tested and deployed for each of these five combinations.

We are committed to only deploying solutions once they are certified. It is highly probable that they won’t all reach the finish line at the same time and deployments will likely be staggered. We know which combination(s) you have and will be prescriptive and surgical in the deployments. 

I cannot set a date for deployment for each combination until beta testing is complete. Once I can, we will issue a notification that deployment has begun, which in every case, we expect to be swift and transparent, with no impact to HOS/ERODS operations.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Thursday, November 7, 2019, 3:00 pm CST

As reported last night, we are recommending drivers with MCP 200, MCP 110 and MCP 50 devices return to electronic logs. As of Thursday morning, more than 99.9% of the impacted devices are transmitting messages and logs and meet compliance requirements – and the percentage continues to climb. The steps we've taken since Saturday night allow for the accurate capture of the driver’s records of duty status and hours of service data (i.e., 10/11, 14/15, 60/70 hours; or 30 minutes) — meeting the AOBRD/ELD usage requirements. 

For drivers that are able to return to electronic logs, there is no need to request an extension of the “8-day paper log” limitation from the FMCSA.

We are actively working to assist the small minority of customers using devices that aren’t transmitting messages and logs. If you believe you might be in this minority, you can find instructions for maintaining Hours of Service compliance here.

We continue to work with a select number fleets who have not received the most recent configuration file because their trucks were out of cellular coverage at the time of start-up. This configuration file doesn’t impact electronic logs but does improve the performance of other real-time applications, including Omnitracs Navigation, Workflow, and Messaging. The number of fleets with this issue also continues to decline as trucks obtain cellular coverage and receive the configuration file.

Lastly, the engineering team is continuing to make good progress in developing a GPS chip remediation plan. This work will ultimately lead to field testing before any declaration that a permanent solution is ready for deployment. We will share progress as it becomes available.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Wednesday, November 6, 2019, 10:13 pm CST

I am pleased to report that 99.9% of all active MCP devices are now transmitting messages and logs containing required compliance information. Therefore, Omnitracs recommends all drivers return to electronic logs, as many impacted fleets already have.

Tonight, we will push an additional update to increase the frequency of communication with the back-office by 2X, which we believe will return performance to pre-event status.

We are here to assist the exceptional few customers who have not received the updated configuration file with getting this accomplished over the next couple of days.

We are in regular communication with the FMCSA and will continue to keep them updated.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Wednesday, November 6, 2019, 11:30 am CST

The Omnitracs team worked tirelessly overnight and I’m happy to report that we continue to make progress.

Last night, we distributed a second configuration file to address devices that were outside of cellular coverage at startup and are encouraged by positive feedback this morning from both drivers and operations teams. I know that some customers are still experiencing issues, and we are working diligently to resolve them.

Many of our customers are reporting near-normal back-office operations, with some returning to electronic logs. We’re now working on a process all of our customers can follow to return to electronic logs while a firmware update is being made available.

I plan to have a material update this evening. I’m targeting 7:00 pm PST/10:00 pm EST and commit to providing information as it becomes available. 

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Tuesday, November 5, 2019, 9:10 pm CST

The Omnitracs team continues to make progress on both the interim fix mentioned in my previous updates — so drivers can return to using electronic logs — and a final, permanent solution. Please feel free to share these updates with your colleagues.

My next update will come tomorrow at 9:00 am PST/12:00 pm EST.

 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Tuesday, November 5, 2019, 4:37 pm CST

 
I’m encouraged to report that we now have correct date-time data and are continuing to see improved system performance following the rollout of the interim config file fix delivered over-the-air.
 
We are, however, seeing a small number of MCP 200 and MCP 110 devices (not MCP 50) that are not able to communicate to or from the host. This issue in communications occurs when the device is outside of cellular coverage at startup. To address this anomaly, we will push an additional over-the-air configuration update this evening. This update will not require a device reset.
 
Drivers and back-office teams are likely reporting that they have seen some functionality return to normalcy. Based on this and other interim steps, we are testing the ability for drivers to return to electronic logs independent of a firmware resolution.

Firmware modifications continue to be triaged with chipset manufacturers. Cellular providers are being kept apprised of the anticipated modifications and are prepared to accept changes when ready. That being said, we are still working to provide a definitive timeline for a permanent firmware resolution. 
 
I will send another update tonight, targeted for 7:00 pm PST/10:00 pm EST.
 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Tuesday, November 5, 2019, 12:00 pm CST

The Omnitracs team continues to work around the clock to solve connectivity issues with MCP 200, MCP 110, and MCP 50 devices.
 
In my last update, I reported that 60% of impacted devices had been updated with a new configuration file addressing date-time data. As of this time, that percentage has climbed to 85%. This configuration file is an interim fix that will display today’s date to the driver, rather than 2000 or 2010. 
 
Last night a patch was also deployed in our operations center that has improved location intelligence for impacted devices on the host (back office). This means that — even though drivers could continue to have a mixed experience throughout their trip  —  back-office applications, including QTRACS and Omnitracs Hours of Service, have significantly increased position and hours of service data to host operations. The patch should provide significant relief in terms of data availability and accuracy to the back office.
 
Additionally, while this issue persists, drivers will need to manually capture odometer readings as they cross state borders for IFTA reporting, should actively check the accuracy of their Record of Duty Statuses on their devices, and maintain paper logs as a backup.
 
In the event an extension of the “8-day paper log” limitation is required for any MCP devices, Omnitracs will take responsibility on your behalf in gaining an extension from the FMCSA and will provide required documentation to you on Friday. This is the most expedient and certain path should it become necessary. 
 
Thank you for your continued patience. 
 
I will send another update at 2:00 pm PST / 5:00 pm EST.
 

Respectfully,

Ray Greer
CEO, Omnitracs, LLC

 

 

Update: Monday, November 4, 2019, 9:13 pm CST
 
Throughout the day, our teams have been pushing a new configuration file to all MCP 200, MCP 110, and MCP 50 devices and subsequently triggering a reset so the changes can take effect. As of this note, 60% of the impacted devices are updated, vastly improving your drivers’ experience and returning applications, except Omnitracs Hours of Service, back to full functionality.
 
While the new configuration file and device reset enable time data to work properly and messages to flow, it is only an interim fix. This temporary fix does not resolve all GPS location data issues. Therefore, drivers should verify electronic logs for accuracy and revert to paper logs when necessary.
 
We continue to work tirelessly in pursuit of a final resolution and are experiencing excellent traction with our chipset partners. We do not yet have a firm ETA for you but expect to deliver greater clarity within 24 hours.
 
If you have any questions or other feedback, please do not hesitate to reach out to any member of your Omnitracs account team. I will provide the next update on this matter tomorrow by 9:00 am PST/12:00 pm EST.
 

Respectfully,

Ray Greer
CEO, Omnitracs, LLC.

 

 

Update: Monday, November 4, 2019, 3:16 pm CST

To provide ongoing transparency and communication regarding the GPS event that occurred at 5pm PT/8pm ET on Saturday, November 2nd, I want to provide an update on our progress and reiterate that resolving this issue is our highest priority. As a reminder, this update only relates to MCP 200, MCP 110 and MCP 50 devices. All other Omnitracs devices are operating as expected.

While our backend systems continue to remain fully functional and maintain bi-directional communications with the devices, we have now deployed an over-the-air configuration file to all impacted units. As units come on air, the configuration file will be enabled once the unit has been reset. Our team is monitoring units and triggering device resets as needed. Our goal is to restore accurate time data to each unit and ensure a significant improvement to the experience of your drivers. This includes the full functionality of all applications except for Omnitracs Hours of Service, which will require driver edits to the electronic logs to maintain compliance.

We continue to work closely with the manufacturers of the GOBI 1000/3000 chipsets used by the impacted units on a firmware update to fix the issue. We will provide an update on our progress with this effort as soon as we have more information. As we continue to address this issue, we advise that drivers continue to maintain paper logs to be fully compliant with ELD/AOBRD regulations.

Our team continues to work around the clock to resolve this urgent matter as quickly and efficiently as possible. While we have not yet achieved a full resolution, please know that this remains our top priority. We will continue to share updates with you on our progress until this urgent matter is fully resolved. We apologize for the continued inconvenience this matter has caused and truly appreciate your patience as we work to address. 

If you have any questions or comments, please do not hesitate to reach out to any member of the Omnitracs team. I will also provide another update on this matter this evening by 7:00 pm PST/10:00 pm EST.

Respectfully,

Ray Greer
CEO, Omnitracs, LLC.

 

 

Sunday, November 3, 2019, 7:51 pm CST

On Saturday, November 2nd, Omnitracs experienced a GPS rollover event that affected our customers' MCP telematics units. Our IVG, Android XRS, and Roadnet devices were not impacted by this event and continue to operate normally. As someone who has spent 30 years in the operational side of the transportation industry, I fully understand the impact this has on operations and the frustration this creates for drivers and their companies.
 
While our backend systems remain fully functional and have bi-directional communications with the devices, we are aware that on some devices, our customers are unable to access both accurate time and location data. Resolving this issue is our highest priority, and our team is working around the clock until all units are again fully operational.
 
In addition to our team's diligent efforts to resolve this issue quickly, we are also working with the manufacturers of the GOBI 1000/3000 chipsets used by the impacted units, to promptly develop a firmware update to fix the issue. Simultaneously, we are exploring software alternatives to address the time and location issues that are impacting the system in the event the firmware update takes longer than expected. In the interim, we advise drivers to maintain paper logs to be fully compliant with ELD/AOBRD regulations.
 
We take our responsibility to our customers seriously, and we know they count on us for accuracy, reliability, and integrity. We value their trust beyond all else. We deeply regret the inconvenience this has caused our customers and will continue to provide regular updates to them as we make progress in fixing this urgent matter.
 

Respectfully,
 
Ray Greer
CEO, Omnitracs, LLC.