xDR Challenge in Manufacturing 2020
xDR Challenge in Manufacturing 2020 is an official off-site indoor-positioning competition track in IPIN 2020 (Track5).
Although IPIN 2020 is canceled,
off-site competition tracks are all held.
Our track is a competition track in industrial scenario. The competitors are asked to estimate trajectories of workers and forklifts by applying their original indoor localization algorithm to the data measured in the factory.
We are accepting pre-admission，now.
Finally, the final registration form is now open and
actual dataset for xDR Challenge 2020 is now ready to provide. After confirming your registration, we will provide the dataset. We are waiting for your registration.
Important Dates
- Result submission due
- 30 November, 2020 (IPIN’s official schedule)
- Winner proclamation:
- 14 December, 2020 (IPIN’s official schedule)
Presentation Slide
xDR Challenge 2020 was successfully finished. The presentation slide for announcing the results and winners can be downloaded from
here.
Overview
xDR Challenge in Manufacturing 2020 (hereinafter referred to as xDR Challenge 2020) will be held as a sequel competition to PDR/xDR Challenge series. Unfortunately, cancelation of main conference of IPIN2020 was announced due to COVID-19. However, we will never give up. We are planning to hold a fully ONLINE off-site competition track as an official competition track of IPIN.
The competition is an off-site tracking competition by sharing sensor data measured in actual industrial fields. The aim of the competition is to encourage the researchers to develop the xDR algorithms which can be utilized for realistic industrial scenarios. The xDR Challenge 2020 will be hosted by PDR benchmarking standardization committee. Last year the targets industrial fields of the competitions are restaurant and factory in manufacturing.
This year, we will focus on manufacturing site as a target field of the competition. The operations of employees can be objectively analyzed for improving the operations by data-driven service engineering.
Similar to xDR Challenge 2018, forklifts are added to be a competition target. Movement of the forklifts are supposed to be tracked by vehicle dead-reckoning. Movement of moving objects (employee, forklifts) can be very important inputs for the objective analysis, improving efficiency, quality of working. In those scenarios, we will prepare two sub-tracks of the competition in the xDR Challenge 2020.
- PDR sub-track: PDR based indoor localization for tracking employees working in the target factory.
- VDR sub-track: VDR based indoor localization for tracking forklifts driven in the target factory.
If you tell us an intention to join the competition, we can provide you previous competition’s dataset as sample data.
As the sample data, we will provide short data measured in the target environment. We can also provide sample datasets of the previous competitions upon request.
We look forward to your participation!
Competition Details
Competition Type: Off-site competition
This competition will be held as an off-site competition for indoor localization technology.
The competition organizers have conducted data measurement with the help of the actual factory
in the manufacturing industry. The dataset measured in the factory will be provided to competitors.
The competitors are required to estimate the targets movement in the factory by applying
their own algorithm and submit the estimated results in the prescribed format.
The submitted results will be evaluated by multi-faceted evaluation metrics based on
the comparison with ground truth.
Target Environments: Factory
This year’s target environment is a factory in manufacturing industry.
This factory is the same factory with the target field of the manufacturing track of
xDR Challenge 2019. The target area is expanded in this year. The detailed floor layout
is shown in figure 1.
Figure 1 Floor layout of the factory
BLE Beacons and Measuring Devices
We utilized Fujitsu's PulsarGum as BLE beacons. The PulsarGum is battery-free beacon. BLE signals are emitted disproportionally, and dependent of the condition of photovoltaics.
The shortest interval of the BLE signal emission is 1.26 sec. The sensor data used for the competition is measured by Biglobe's BL-02.
Competition Tracks
-
PDR sub-track:
The competitors are required to estimate the movement (trajectories)
of the employees working in the factory by using their original indoor localization algorithm.
We believe that PDR-based indoor localization algorithm is a suitable for tracking the emplyoees in the wide-area factory as targeted in this competition.
In this sub-track, we provide the sensor data required for the PDR-based indoor localization algorithm as the dataset of the off-site competition.
We asked the employees to bring Android device to collect sensor (gyro, accelerometer, magnets sensor,
barometer) data as well as the BLE signals from the BLE beacon.
Partial positional reference will be provided for positional correction.
The competitors are required to submit results of the indoor localization.
We will evaluate the submitted results by our multifaceted evaluation metrics.
-
VDR sub-track:
The competitors are required to estimate the movement (trajectories) of
the forklifts driven in the factory by using their original indoor localization algorithm.
We believe that VDR-based indoor localization algorithm is a suitable for tracking the forklifts in the wide-area factory as targeted in this competition.
In this sub-track, we provide the sensor data required for the VDR-based indoor localization algorithm as the dataset of the off-site competition.
We attached Android device onto the forklift to collect sensor
(gyro, accelerometer, magnets sensor, barometer) data as well as the BLE signals from the BLE beacon.
Partial positional reference will be provided for positional correction.
The competitors are required to submit results of the indoor localization.
We will evaluate the submitted results by our multi-faceted evaluation metrics.-
Evaluation Framework
We are standardizing evaluation of indoor localization especially for algorithms based
on xDR in the PDR benchmark standardization committee.
The evaluation framework applied in this competition was determined according to the discussion
in the committee.
PDR benchmark standardization committee (PDRBMSC) is discussing and proposing standard evaluation
framework for evaluating indoor localization methods or systems.
The PDRBMSC proposes three evaluation indicators and negative check criteria.
In this track, we adopt the PDRBMSC’s evaluation frameworks.
The final evaluation metrics and integrated index will be calculated with
these indicators and negative checks. The brief introduction of
the evaluation indicators and negative checks is as follows;
Evaluation indicators (Tentative)
- Circular Error (CE) : Absolute 2D error compared with ground truth points.
-
Circular Error Distribution Deviation (CEDD) Circular Accuracy: Evaluating distribution of error in 2D error space.
Distance between the peak in the error space and origin (0,0) will be evaluated.
- Error Accumulation Gradient (EAG): Evaluating speed of error accumulation from correction
points when true positions are limitedly provided.
Negative Check Criteria (Tentative)
- Requirements for Moving Velocity: checking if moving velocity is within the decent range.
- Requirements for Validity of the Trajectory: checking if points consist of trajectory are in valid area
In addition to our evaluation indicators, CE75 will be provided as a reference, for consistency with the
other IPIN competition Tracks. Note that the CE75 is NOT used for determining the winners. The evaluation frameworks and scripts for calculating final scores for determining
the winners is now available from the following our GitHub account. The evaluation scripts can
calculate CEs with arbitrary percentiles.
How to Participate
Step1 Request for Admission
If potential competitors have an interest in our competition.
Please register pre-admission in our pre-registration form.
We will provide you our sample after we confirm your registration
Step2 Downloading Sample Data
We will provide link to our sample data for those you completed pre-admission.
We also provide the script for evaluating the results via GitHub.
Step3 Application the Competition
If you decide to join the competition, please register final admission from our web form.
We will provide you our real test data after we confirm your registration.
Step4 Submission of the Technical Description
This competition aims to compare indoor algorithms and to provide guideline for the indoor
localization users. Therefore, we need to survey the competitors’ algorithm. To this end,
we will ask you to submit technical description about your indoor localization algorithm.
The format for the technical description will be announced soon.
Step5 Payment of subscription fee
Registration fee is 100 Euros for the first IPIN's competition Track plus 50 Euros for any additional Track, to be paid when results are submitted.
Track5 has two sub-tracks. Track5 charges only 100 Euros (50 Euros for aditional track) for one team
if the team join both sub-tracks.
PDRBMSC is planing to financially support for subscription fee for those who meet our requirement.
If you desire to be supported by PDRBMSC, please contact us.
Step6 Result submission
Competitors should submit their result of estimation of the trajectory with pre-determined
format which includes x,y coordinate of the target positions and orientation of the targets with timestamps.
Definition of exact format for submission will be included in the actual test data package.
Dead-line for result submission: 30 November 2020
Step7 Announcement of the results
The result of the xDR Challenge 2020 will be announced on-line.
Prize
Thanks to our sponsors, we will provide gorgeous prize including cash prize and prize item for the winners of the both tracks as follows:
PDR sub-track:
Winner: [JPY150,000] or [JPY100,000 + SSEI PDR-W]
VDR sub-track:
Winner: [JPY150,000] or [JPY100,000 + SSEI VDR Module (SUC-VDR100)]
Prize items
SSEI’s PDR-W
SSEI’s VDR Module
Thanks to our sponsor Sugihara SEI (SSEI), the winner can get the prize item, if the winner chose. SSEI provides dedicated modules for PDR and VDR
Reference
- Takeshi Kurata, Takashi Maehata, Hidehito Hashimoto, Naohiro Tada, Ryosuke Ichikari, Hideki Aso, Yoshinori Ito: IoH Technologies into Indoor Manufacturing Sites，Proceedings of Advances in Production Management Systems Conference, 2019
- Ryosuke Ichikari, Katsuhiko Kaji, Ryo Shimomura, Masakatsu Kourogi, Takashi Okuma, Takeshi Kurata: Off-Site Indoor Localization Competitions Based on Measured Data in a Warehouse, Sensors, vol. 19, issue 4, article 763, 2019.
- Takeshi Kurata, Ryosuke Ichikari, Ryo Shimomura, Katsuhiko Kaji, Takashi Okuma, Masakatsu Kourogi: Making Pier Data Broader and Deeper: PDR Challenge and Virtual Mapping Party, MobiCASE 2018 (9th EAI International Conference on Mobile Computing, Applications and Services), 2018.
- Masakatsu Kourogi and Tomohiro Fukuhara: Case studies of IPIN services in Japan: Advanced trials and implementations in service and manufacturing fields in special session "Value Creation in LBS (Location-Based Services)", IPIN 2017.
Organizing Committee
- General Co-Chair: Ryosuke Ichikari, Ph.D., AIST, Japan
- General Co-Chair: Ryo Shimomura, AIST & Univ. of Tsukuba, Japan
- Dataset Chair: Nozomu Ohta, AIST & Univ. of Tsukuba, Japan
- Evaluation Chair, Satsuki Nagae, AIST & Univ. of Tsukuba, Japan
- International Liaison Co-Chair, Antonio Ramon Jimenez Ruiz, Ph.D., CSIC-UPM, Spain,
- International Liaison Co-Chair, Soyeon Lee, Ph.D., ETRI, Republic of Korea
- Industrial Liaison Chair, Takeshi Kurata, Ph.D., AIST & Univ. of Tsukuba, Japan
Contact
e-mail: M-xDR-Challenge2020-ml[AT]aist.go.jp
