CES 2024：自動運転エンジニアからの総括

CES 2024: Wrap-up from an autonomous driving engineer

CES 2024 was an exciting week in Las Vegas for the TIER IV team. We showcased multiple products and demos, introduced new global projects, and conversed with people across the industry. I’m Yu Asabe, an autonomous driving (AD) software engineer at TIER IV, and in this tech blog, I would like to share some of the highlights of my experience at this year’s Consumer Electronics Show.

Beautiful morning at the Las Vegas Convention Center’s West Hall

Strolling through the TIER IV Booth

Just as Day 1 of CES started, TIER IV’s spot in the West Hall was already attracting quite a bit of attention, with the booth size three times bigger compared to last year and a total of five live demos being showcased. Overall, our booth was separated into three sections: one side displaying one of our concept Robobus vehicles for our white-label Level 4 AD solution, “fanfare”, along with displays describing our Pilot.Auto and Web.Auto products. The opposite side was filled with demos of our Edge.Auto product, and the middle section served as a comfortable meeting space and general inquiries counter.

The Robobus was flanked by screens displaying information about Pilot.Auto, Web.Auto and “fanfare.”

The futuristic-looking Robobus was the center of attention on this side of the booth, with our team members providing visitors with explanations of our white-label mass production-ready vehicle solution “fanfare,” and our Pilot.Auto and Web.Auto product lineups.

Pilot.Auto is a full-stack AD software platform that is both scalable, microautonomous, and fully customizable. It is based on the open-source AD software stack called Autoware, and we at TIER IV enable our customers to develop all sorts of AD software solutions by building on top of this open-source project. Autoware includes everything from localization, sensing, perception, and planning to vehicle control. Pilot.Auto also comes with software tools and drivers for easy hardware integration. It is essentially a well-verified and validated version of Autoware that delivers the performance reliability necessary for commercial operations. Depending on different use cases, requirements, and operational design domains (ODDs), we can tailor Pilot.Auto to make it the perfect fit for each of our customer’s services.

In 2023, TIER IV received Level 4 certification for its Pilot.Auto-based AD system, “AI Pilot”, from the Japanese government. Being the first in Japan to receive this certification for a full-stack AD system, we are now able to use our years of experience and knowledge to further pursue Level 4 autonomous driving in more complicated ODDs across the country.

When developing and operating AD solutions for commercial services, the efficiency and reliability of the development process and operation of the vehicle fleets become critical issues. For this, we have our cloud-native DevOps platform Web.Auto, which is a bundle of tools that streamline the development and operation of autonomous solutions.

The Web.Auto Evaluator tool is a CI/CD platform and MLOps toolchain that manages the test-driven development of software and ML models. Developers can create scenarios and run simulation-based tests that verify the features and functions of the AD software.

During the actual operation of vehicles, the Fleet Management System can be used to monitor the state of the vehicles in real time, checking their route or schedules and analyzing potential warnings in the system. Remote monitoring allows operators to continuously check the safety of the vehicle operation and to intervene in emergencies.

Reliable and privacy-preserving data collection is also important for the continuous improvement of the AD solution, where machine learning models can be re-trained, and post-operation data analysis can lead to further improvements in the software.

For more information about our various products and solutions, “Unraveling TIER IV’s autonomous driving solutions” is a good place to start.

During CES this year, TIER IV launched Edge.Auto, a reference platform designed for the rapid development of AD systems that includes sensor and ECU hardware components along with useful software tools and open-source sensor drivers. We had several demos at our booth that showcased our newest products along with their innovative capabilities.

Edge.Auto product demos filled the other half of the booth, with TIER IV members busy showcasing the lineup

One of them showed our in-house developed automotive cameras in action, comparing the performance of our C1, C2, and newly developed C3 cameras. Offering resolutions from 2.5MP to 8.3MP and a dynamic range of 120dB, these cameras are suitable for both advanced driver-assistance systems (ADAS) and AD applications, and other use cases requiring highly capable automotive-grade cameras. These cameras use a GMSL2 interface and come with features such as LED flickering mitigation.

Traffic light recognition with the C2 camera

Another demo showcased the interface conversion module that leverages ADI’s GMSL2 (Gigabit Multimedia Serial Link Generation 2), and converts streams of up to eight cameras into 10Gbit Ethernet. Not only is Ethernet widely available on more types of compute hardware as a standard interface, additional features including synchronization with other sensor devices such as LiDAR through PTP, shutter timing control, and image-data preprocessing/timestamping are also possible, making the module suitable for a wide range of complicated perception tasks.

Research engineer David Wong runs through a demo.

We also had an embedded Edge AI accelerator demo that used a camera and LiDAR as sensor inputs and a 2.5W Hailo-8 accelerator board to conduct inferencing for semantic segmentation. Multi-modal semantic segmentation was conducted on both the camera and LiDAR inputs individually, and also the fused data. It was able to show a level of sensor redundancy, where if either the camera or LiDAR input was blocked (by some obstruction for example), the semantic segmentation of the pedestrians continued to operate. On the first day of CES, we collected additional data at the booth and team members back in Japan used the data to train the model further to improve the demo performance.

A LiDAR-camera sensor fusion demo was conducted on the center tower, with live results displayed on large screens. At the top of the tower were three sets of a C2 camera and LiDAR, which were provided by different vendors. All the sensors in this demo used our universal sensor driver and perception system, and the positions were spatially and temporarily calibrated using the C2 cameras. The main task at hand here was pedestrian perception and tracking using edge ECUs. As people moved about the booth, you were able to see them recognized and tracked as they moved across the FoV of each of the LiDAR.

Another big announcement TIER IV made during CES 2024 is the initiation of the Co-MLOps (Cooperative Machine Learning Operations) Project that allows for the scaling of AI development for autonomous driving by sharing globally sourced sensor data and offering MLOps functions and Edge AI reference models to partner companies. Conventionally, companies have individually collected data and engaged in developing similar types of technology in order to collect the large datasets necessary for generating competitive AI models. ML expertise and data were all black-boxed, with the market being dominated by a few big companies. Smaller players in the industry (and startups) with limited resources have significant disadvantages in this data-collection and model-generation process, which has negatively impacted the scalability of technological development over the years. This new initiative, Co-MLOps Platform, aims to serve as the foundation for various companies and research institutions to pitch in and foster open innovation through open collaboration. It is a global platform where ML expertise and data can be shared, and the advancement of AD perception technologies can be harmonized and elevated collaboratively. All the data collected will be shared with appropriate privacy and security safeguards, and MLOps functionalities on these large-scale datasets will be shared among the members.

Members of the platform will have access to the shared dataset, with the ability to create baseline AI models at low operational expenses. Using the results of the platform as a starting point, everything from there on out can be done privately, enabling all members to set themselves apart.

The resulting output from the Co-MLOps Platform can then be integrated into Software-Defined Vehicle projects such as the Open AD Kit (more on this later) to realize more advanced SDV-based autonomous driving solutions.

Software-Defined Vehicles (SDV)

Throughout the automotive space this year, there was a great deal of focus on Software-Defined Vehicles (SDVs). The industry is seeing a shift toward a hardware-agnostic, software-centric approach for various vehicular applications and functions. One key component that received attention at the TIER IV booth is using Autoware as an application on top of the SOAFEE framework. SOAFEE, or the Scalable Open Architecture for Embedded Edge special interest group (SIG), is an industry-led collaboration between companies across the automotive and computing space to build together an open-source architecture for software-defined vehicles. The goal is to “create a shared platform for vehicles using a cloud-native architecture that accommodates multiple hardware configurations.” By stimulating industry collaboration to develop new software architectures and methodologies, new industry standards for the base vehicular hardware and software can be formed, which will maximize reusability and speed of delivery of more advanced AD and SDV applications. ADAS, in-vehicle infotainment, autonomous driving solutions, and innovative vehicular functions are all considered competitive areas that can create differentiated value.

The Autoware Foundation is working on the Autoware Open AD Kit, which brings the best practices of SDV development to the Autoware ecosystem and provides optimized hardware and software solutions. Automotive software and applications will be developed with a cloud-native approach, enabling software to be developed, tested, and validated on the cloud before being pushed to fleet vehicles through over-the-air (OTA) updates. Working closely with the Arm Automotive Platform, the Autoware Open AD Kit can be run on an Arm processor in the cloud and on the vehicle, providing a seamless transition from the cloud into the vehicle during deployment. This can unleash the power of the shift-left development methodology that can allow the ecosystem to develop and verify software before vehicle hardware platforms are even available.

Open AD Kit Blueprint

During CES, we had a live demo with Arm that explained how we can deploy a new AD feature in a cloud-native way using containerized Autoware components and the Open AD Kit platform. We considered a scenario where a behavior change was necessary on the planner component of Autoware. Originally, when an object was detected in the ego-vehicle’s lane, the ego-vehicle would come to a full stop. However, we wanted to change the planner so that when an object is detected in the ego-vehicle’s lane, the ego-vehicle changes lane and proceeds forward.

After the appropriate changes to the code have been made, three steps are necessary to complete the SDV validation and deployment cycle. First, functional validation is performed in the cloud through the simulator provided by TIER IV’s Web.Auto Evaluator solution. When the desirable function is validated, the validated planning container then moves to the virtual vehicle, which is a cloud-based SDV that has ISA-level parity to the edge hardware on the vehicle. After another round of simulation here