xDR Challenge2023
~PDR and BLE with LiDAR based GT~

xDR Challenge is a series of indoor localization competition by PDR benchmark standardization committee mainly focused on integrated indoor localization methods with dead-reckoning. xDR Challenge 2023 will be held as an official competition track of IPIN 2023 (Track 5). Our track is categorized as "off-site, off-line" track, which means that we provide datasets for indoor localization to competitors and the competitors are asked to submit estimated trajectories of targets in certein time limit. Target field of this year’s competition is commercial facilities and the targets of localization are pedestrian walking in the field. We arranged totally about 100 BLE beacons in the field for correcting error of PDR. We adopted LiDAR for collecting ground-truth of position of the targets. We will evaluated the estimated trajectories of the target by utilizing multi-faceted evaluation metrics with the precise ground-truth data by LiDAR.

(URL:)https://unit.aist.go.jp/harc/xDR-Challenge-2023/index.html

Definition of data and trials used in this track

Training data, which are provided as off-line files and are used to calibrate the system (provided at Dataset section).

Testing trials, which are provided through the EvaalAPI and are reloadable. Competitors can run them as many times as they like to evaluate and fine-tune their system as well as to get used to the EvaalAPI.

Scoring trials, which are provided through the EvaalAPI and are only available at specific times during the competition production. Those are used to evaluate the competitors. Some of the scoring trials include data from a subject not in the training data.

* These are based on definitions in Track 3 of IPIN Competition with slight modification for our track.

• (Sept. 8th, 2023) Evaluation details are added. Evaluation tools are shared script on github
• (Aug. 29th, 2023) FAQs about the scoring trials is added.
• (Aug. 28th, 2023) E-mail address to submit the technical description is added. (Step3 of "How to Participate")
• (Aug. 25th, 2023) Announcement for the official application and registration are added (in "Important Dates" and "How to Participate")
• (July 21th, 2023) Unveiled the prize for the winner
• (July 12th, 2023) Included gyro's drift information from Android in the training data. Also, an example of PDR, integrated localization has been added.
• (July 3rd, 2023) Included geomagnetic information of the facility in the readme of the training data
• (June 26th, 2023) Started providing the entire training data for those who registered pre-registration
• (June 6th, 2023) Started providing the first sample data for those who registered pre-registration
• (May 26th, 2023) Started accepting pre-admission on our pre-registration

xDR Challenge 2023 ~PDR and BLE with LiDAR based GT~ (Hereafter referred to as xDR Challenge 2023) will be held as a sequel competition of PDR/xDR Challenge series which had been hosted by PDR Benchmark standardization committee. As in previous years' competitions, the xDR Challenge 2023 will be held as an official competition tracks of the IPIN conference. There are three categories of tracks in IPIN 2023 competition. Our track is categorized as an "Off-site, Off-line" track.

The off-site competition means that
- Competition organizers have conducted measurement and prepare dataset for the competition, and
- The dataset measured will be provided to competitors, and
- The competitors are required to estimate the targets movement by applying their own algorithm and submit the estimated results.
The off-line competition means that
- the competitors are not required to submit the results in real-time, but to submit in certain (longer) time limit, and
- the competitors can obtain whole data of each sequence for applying global optimization while constraining with provided information.

The datasets of our track consists of sensor data required for the PDR-based indoor localization. The data are measured in the commercial facilities in a highway rest spot. We measured pedestrian movement by using Android devices. We collected sensor (gyro, accelerometer, magnets sensor) data as well as the BLE signals from the BLE beacons.

Similar to the previous PDR/xDR Challenges, submitted trajectories will be evaluated by multi-faced evaluation metrics. It is notable that we collected highly precise ground-truth of the targets movements by using a hand-held LiDAR. The detailed evaluation metrics will be announced at a later date.

xDR Challenge 2023 will be conducted closely together with other tracks of the IPIN competitions under a common schedule. We also adopt common tool named EvAAL API for sharing dataset and receiving results. As aligned with other tracks, real competition will be conducted at a day in the middle of September 2023. We will provide training data and testing trials in same format with the real dataset for allowing the competitors to prepare and adjust localization algorithm/systems before the real competition.

We look forward to your participation!

Target Environments: Commercial facility

This year's target environment is a commercial facility. This facility is an express way service area. It has two buildings; one is two-story and the other is one-story. We utilized MyBeacon (Aplix) as BLE beacons. BLE signals are emitted every 0.1 sec. The beacon locations will be provided as (x, y, z) location in coordinates of the floor map.

Measuring Devices and Data Distribution

9-axis IMU sensor data used for the competition is measured by AQUOS Sense 6 (SHARP). The ground truth of the location was measured with a hand-held LiDAR (GeoSLAM ZEB-Horizon) at approximately 100 Hz. The ground truth will be disclosed only for training data. The data will be distributed over EvAAL API. Overview of the provided data are shown in Table 1.

Table 1. Overview of the data

Requirements for competitors

The competitors are required to estimate the movement (trajectories) of the subjects walking in the facility by using their original indoor localization algorithm. We believe that PDR-based indoor localization algorithm is a suitable for tracking the subjects as targeted in this competition. We provide the sensor data required for the PDR-based indoor localization algorithm as the dataset of the off-site competition. We asked the subjects to bring Android device to collect sensor (gyro, accelerometer, magnets sensor) data as well as the BLE signals from the BLE beacon. The competitors are required to submit results of the indoor localization (time-stamped x, y coordinates). We will evaluate the submitted results by our multifaceted evaluation metrics. In order to evaluate the performance of PDR only (EAG, etc.), we will ask competitors to submit evaluation data in two phases as follows;

Evaluation Framework

We are standardizing evaluation of indoor localization especially for algorithms based on xDR in the PDR benchmark standardization committee. The evaluation framework applied in this competition was determined according to the discussion in the committee. PDR benchmark standardization committee (PDRBMSC) is discussing and proposing standard evaluation framework for evaluating indoor localization methods or systems. The final evaluation metrics and integrated index will be calculated with indicators and negative check. The evaluation scripts used for the competition are now available on PDRBMSC's github repository. The brief introduction of this year's evaluation indicators and negative checks are as follows;

Table 2. Overview of evaluation indexes

Formula for evaluating indexes and the integrated index for determining winner

The winner of the competition is determined by the integrated index I_i which is calclulated by weighted-sum of the indexes. The formulas for elemental and the integared indexes are shown in Table 3. The indexes are ranging within 0 to 100. The indexes are linearly changing between conditions of max. index and min. index. The weights for calculating I_i are shown in Table 4.

Table 3. Formulas for elemental and integrated indexes

Table 4. Weights for elemental indexes

Frequency of the evaluation

Note that frequency of the evaluation depends on the frequency of the ground-truth data. The frequency of the ground-truth data for xDR Challenge 2023 is about 100Hz. If the sampling frequency of your estimation is less than 100Hz, your estimation can not be accurately evaluated. We recommend you to estimate trajectories in 100Hz or to up-sample the trajectories to 100Hz.

In addition to our evaluation indicators, CE75 will be provided as a reference, for consistency with the other IPIN competition Tracks. Note that the CE75 is NOT used for determining the winners. The evaluation frameworks and scripts for calculating scores for determining the winners will be provided from our GitHub account. The evaluation scripts can calculate CEs with arbitrary percentiles.

Step1 Request for Admission

If potential competitors have an interest in our competition. Please register pre-admission in our pre-registration form. We will provide you our sample after we confirm your registration

Step2 Downloading training data and try testing trials

We will provide ID and password to our training data for those you completed pre-admission. You can download the dataset with provided ID and password. After downloading the sample data, the competitor can start development. Also, we will provide opportunities to run testing trials through EvAAL API with the servers hosted by competition organizers.

Step3 Application of the Competition (Deadline Aug. 31, 2023)

If you decide to join the competition, please register final admission in EvAAL's official application form. The competitors are required to provide short and long descriptions of the system. Please describe required inforamtion in the application form and send a short (2-4 pages) Technical description of your localization system by e-mail to the track5 organziers' ML (M-xDR-Challenge2023-ml@aist.go.jp)

Step4 Registration of the conference and payment of registration fee

In addition to the application, the competitors are required to complete conference registration and pay the registration fee. Please complete your registration and payment of the fee in the registration page. If competitors only join the off-site competition track, there is the designated type of registration for them.

Step5 Scoring trials

On a track-specific day during the second week of September, the competitors are required to join the scoring trials. We will provide unreferenced data for scoring trial through EvAAL API. Competitors will be asked to download sensors data all at once and have a limited time to upload all estimates at once. We will have following two scoring trials one after another. See the requirements for competitors for more detail.

Step6 Announcement of the results

The result of the xDR Challenge 2023 will be announced during the IPIN 2023 conference

Training data (last update : 2023 July 12th)

ID and password are provided after pre-registration.

README

README (Japanese)

Figures of README

Data

Frequently Asked Question about the Scoring Trials

Q.Who is the subject of the scoring trial?

A. The subject numbers for the scoring trials will not be disclosed before the competition. The scoring trials contain both subjects included in the training data and those not included.

Q.Which terminal was used for measuring dataset of the scoring trial?

A. The scoring trials include only terminal number 51.

Q. Can we get calibration data that the subjects stand still ?

A. We did not intentionally measure such calibration data. At the start and end of the data, there might be periods with no activity. These periods might be included in the Scoring trial, but are not guaranteed.

e-mail: M-xDR-Challenge2023-ml@aist.go.jp