Track 4: "PDR for warehouse picking (off-site)"

Challenge Goal

The goal of this challenge is the precision improvement of positioning technique, sharing a common recognition of problems and establishment of a method to compare the warehouse technique for the logistics by competing the precision of the pedestrian dead reckoning(PDR) techniques using the picking data measured in a real warehouse for the logistics. Comparing to the usual walking, PDR positioning using the measuring data in a warehouse includes a variety of tasks like picking, gathering, walking in a warehouse, carrying goods with carts and so on. Therefore, the advanced PDR positioning techniques and its correction techniques to treat a variety of situations as mentioned above are needed.

Results

Summary of PDR Challenge in Warehouse Picking (IPIN 2017 Competition Track 4)

Prize

The winner of Track 4 will be awarded one of the following prizes by PDR benchmark standardization committee (JP) according to the winner's preference.

1) PDR module (Sugihara SEI Co. &Lt., 200,000 JPY worth) + 100,000 JPY

2) 150,000 JPY

Main feature of the challenge

Off-site challenge approach　

This challenge is done off-site, so all data processing will be done using sensor data measured by 8 people in a warehouse for about 3 hours for each. For evaluation standard, competitors need to use the WMS picking logs recorded during measuring the sensor data.

Multiple sources of information

As the sensor data for PDR, the gyro data, the acceleration data, the magnetic data, the pressure data and BLE beacon data were measured using the android smartphones (LG Nexus5). The smartphone was mounted by the warehouse worker and the wearing position was kept as same as possible.

Continuous motion and recording process

The test subjects of this challenge were the workers picking goods in a warehouse. The expected actions of the picking workers are as follows.

Walking in a warehouse
Vouching
Picking at shelves (including the operating the terminals)
Carrying goods using carts

Information about correct actions of each time are not given.

Realistic walking style

As mentioned above, during data measurement, the test subjects do not always walk. They do several types of behaviors while changing their position. The selection of the moving route for the picking instruction was left up to the workers in a warehouse, so while walking a variety of moving ways are possible like stop on the route, change the direction, side stepping and rear walking.

Phone holding

The smartphone is fixed to the waist with a waist pouch as shown in the figure.

Desired localization approaches

By the nature of the data measured in this challenge, the positioning methods used by the contest participants are desired to have following abilities.

PDR method which has robust positioning functions for moving in a warehouse
Functions to cancel out accumulated errors by given discrete positional refeference and RSSI of BLE beacons
Time series optimization using the given data
Keeping naturality of attitude during the picking action (picking position, amount of moving)
Keeping naturality of walking speed
Function to avoid incursion into the area of objections

Warehouse

Logistics warehouse A

Click the image to see the enlarged image

Logistics warehouse B

Click the image to see the enlarged image

Description of Datasets

Description of Competition data

Warehouse A is sample data showing long-term measurement in the logistics warehouse.
Warehouse B is data to be used in the real competition, and sample data of Warehouse B extra only 30 minutes from the measurement start time.

The time (in Unixtime) of the range of the measurement in Warehouse B of this competition is as follows.

Terminal_ID,Measurement_start_time,Wearing_start_time,Measurement_end_time

1,1485239145,1485240508,1485252057
2,1485239148,1485240513,1485248722
3,1485239234,1485240524,1485252460
4,1485239243,1485240545,1485252313
5,1485239286,1485240550,1485252407
6,1485239291,1485240555,1485252344
7,1485239328,1485240562,1485252227
8,1485239334,1485240568,1485252085

"Terminal_ID" is a number for distinguishing terminals. "Measurement_start_time" is the time when the terminal started recording sensor data. "Wearing_start_time" is the time when the terminal was actually worn on the employee's body. Before the workers worn the terminals, the terminals were arranged on a table. "Measurement_end_time" is the time when the terminal stop recording at the measurement.

Actual evaluation points for real competition are existed between wearing_start_time and measurement_end_time.

We are disclosing the data which is extracted only first 30 minutes of warehouse B as sample data. Data extraction was performed on "Sensor data of terminal(Working data)" and "WMS". Note that sample WMS data include WMS logs before Measurement_start_time, you can ignore the data outside of the range of measurement.

Dataset list for logistics warehouse A

Sensor data of terminal

Note: The data format provided is what we defined, but since smartphones are used for measurements this time, the unmeasured sensor data is included in our data format. Specifically, the values of gyro temperature (xyz) and air temperature are written as '0' , but they are ignorable.

: Sensor data when the terminal is placed on a stationary flat surface (.txt)
: Format; Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
: Sensor data when writing a figure 8 in the air with the terminal in hand (.txt)
: Format; Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
: Sensor data when picking with the terminal attached (.txt)
: Format; Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t

Map information

: Map image (.png)
: Map size (.csv)
: Format; The coordinates of the four corners of the map and the coordinates at the start (xyz)
: Shelf arrangement (.csv)
: Format; Shelf number, Arrangement coordinates (xyz), Width, Depth, Direction to pick

WMS data

: Picking operation (.csv)
: Format; Worker number, Time picked, Shelf number, Number of picks

Dataset list for logistics warehouse B

Sensor data of terminal

: Sensor data when the terminal is placed on a stationary flat surface (.txt)
: Format; Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
: Sensor data when writing a figure 8 in the air with the terminal in hand (.txt)
: Format; Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
: Sensor data when picking with the terminal attached (.txt)
: Format; Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t, BLE reception information

Map information

: Map image (.png)
: Map size (.csv)
: Format; The coordinates of the four corners of the map and the coordinates at the start (xyz)
: Shelf arrangement (.csv)
: Format; Shelf number, Arrangement coordinates (xyz), Width, Depth, Direction to pick

WMS data

: Picking operation (.csv)
: Format; Worker number, Time picked, Shelf number, Number of picks

BLE beacon information

: BLE beacon information
: Format; MAC address, Installation coordinates (xyz)

How to participate?

Competitors are requested to follow the steps as mentioned below for participating this challenge.

Step1 Request for admission

During the registration period, contestant of the competitions can register themselves as the competitors of the PDR challenge.

　For this track 4, admission process can be separated into pre-admission, and real-admission.
If you have an interest for joining PDR challenge, please send an e-mail to pdr-warehouse2017@aist.go.jp with your self-introduction (at least with your name and institution). Then we will provide the access pass (ID and Password) for downloading sample data. We regard this e-mail as your pre-admission for track 4.
Real-registration follows the same Admission Process with the other tracks.

[Admission Process]
A "competitor" can be any individual or group of individuals working as a single team, associated to a single or a number of organizations, who wants to participate in one or several tracks.

Competitors apply for admission to the competition tracks by providing a short (2 to 4 pages) "technical description" of their localization system, including a description of the algorithms and protocols used. The technical description must be sent by e-mail to the chairs of the intended track. Track chairs will accept or refuse the application in a short time, based on technical feasibility and logistic constraints.

After acceptance of the competitor's technical description, one member of the competing team is required to register to the IPIN conference, specifying that the registration is linked to a competition track. Full registration to the conference covers participation in the competition process (allocated time, support and space) and the submission of a paper describing the system.

Competitors are required to make an oral presentation of their system during a dedicated session at the IPIN conference. Additionally, competitor teams are invited, but not required, to submit a paper to the conference. The paper will follow the same peer review and publishing process as all the other papers submitted to IPIN. If the paper is work-in-progress, competitors can benefit from relaxed deadline as specified below.

Competitors who wish to test their system, but do not want to be included in the public rankings and compared with other competitors, can ask for that special condition in advance to the chairs of the relevant track.

Step2 Downloading sample data

　If the pre-admission for the PDR challenge is completed, competitors can obtain a password and a user ID for downloading the sample data of the PDR challenge so that they start preparing the competition.

Step3 Registration of IPIN conference

　If you decide to join the track 4, you are required to complete real-admission. In order to complete real-admission, you need to submit a short (2 to 4 pages) "technical description" of their localization system, including a description of the algorithms and protocols used. And you (at least one member from your group) are also required to register to the IPIN conference.
Sample data provided for whom they completed pre-admission are not the final data for the competition. Actual test data and participation rights for the competition are exclusively provided for competitors who completed also completed the conference registration of the IPIN2017.

Step4 Access to the test data for final evaluation

Actual data for the competition are provided after confirmation of the completion of the conference registration. The competitors can obtain another user ID and a password for downloading the actual data for the competition.

Step5 Result submission

Competitors should submit their result of estimation of the trajectory with predetermined format which includes x,y coordinate of the target positions and orientation of the targets with timestamps. Definition of exact format and sample data for submission will be included in the actual data package described above.
Dead-line for result submission: 8 September 2017

Step6 Oral presentation of results at IPIN Conference

Competitors are required to present their methods and algorithms, and state their advantages of the methods and algorithms in the light of the evaluation criteria in the special session of the competitions during the conference. Additionally, competitors are invited, but not required, to submit a paper to the conference. At the end of the session, the competition organizers will show competitors' results and estimated trajectories. The results will be scored by a composite indicator according to the evaluation criteria.

Evaluation metrics

Evaluation index

Integrated positioning error evaluation (E_d) Score 0~100
PDR error evaluation (E_s) Score 0~100
Picking work evaluation (E_p) Score 0~100
Human moving velocity evaluation (E_v) Score 0~100
Obstacle interference evaluation (E_o) Score 0~100
Update frequency evaluation (E_f) Score 0~100
Comprehensive evaluation (C.E.) Score 0~100

Note: The WMS provided as test data is a combination of two data. First is WMS of sample data (picking data for 30 minutes from the beginning). Second is WMS of true position decimated (picking data after 30 minutes).
The section to be evaluated is unified to the estimated trajectory excluding the sample data time.

Integrated positioning error evaluation (E_d)

E_d is Comparative evaluation with true position. This is an index to evaluate the entire estimated trajectory obtained by utilizing all the data provided. The median of all the position errors is the value which becomes the median value based on each position error (Fig.1).
Evaluation method: The score is calculated from the median of all position errors (Fig.2).

Fig1. Relationship between estimated trajectory and true position

Fig2. E_d score as function of the median of position error

PDR error evaluation (E_s)

E_s is Comparative evaluation with true position. This is an index to evaluate the error accumulation of PDR occurring between correction positions (WMS).
Evaluation method : The regression line of the position error is calculated based on the elapsed time from the correction position (Fig.3). The score is calculated based on the slope of this regression line (Fig.4).

Fig3. Relationship between position error and time

Fig4. E_s score as function of the slope of regression line

Picking work evaluation (E_p)

E_pis Comparative evaluation with true position. This is an index to evaluate likelihood of picking work. Evaluate whether the picking operation at the true position (decimated WMS) can be guessed or not (Fig.5). Each picking operation is done within a minimum of 3 seconds from the update time of the true position. In the case of picking work, the workers rarely walk.
Evaluation method :The score is calculated from the number of times you can check walking stop (tolerance 1 [m]) for 3 seconds at the true position (Fig.6). Refer to (1) for derivation formula.

Fig5. Picking operation example in true position

Fig6. P_i score as function of the movement distance for 3 seconds

P_i: Number of times picking determination was made*. P_max : Number of all true positions.
*Count when walking stop (tolerance 1[m]) for 3 seconds is confirmed at the true

Human moving velocity evaluation (E_v)

E_v evaluates the entire estimated trajectory. This is an index to evaluate the humanity of the moving velocity. It is evaluated whether the estimated trajectory deviates from the human moving velocity by the influence of the correction position or the like (Fig.7).
Evaluation method: The velocity is calculated based on the distance and time between each estimated position. The score calculates the corresponding number of times from within the estimated trajectory with the moving velocity of the human being less than 1.5 [m / s] (Fig.8). Refer to (2) for derivation formula.

Fig7. Relationship between movement velocity and walking velocity threshold

Fig.8 v_i score as function of the moving velocity

v_i: Number of times that it is determined to be a human moving velocity*.v_max: Number between all trajectory positions.
*Count when moving velocity is less than 1.5[m/s].

Obstacle interference evaluation (E_o)

E_o evaluates the entire estimated trajectory. This is an index to evaluate whether the estimated trajectory interferes with obstacles. Evaluate whether trajectories interfere with obstacles such as walls, pillars and shelves (Fig.9).
Evaluation method: The score is calculated according to the interference distance with the obstacle by linearly interpolating the estimated position (Fig.10). The allowable error of obstacle interference is 0.17 [m]. Refer to (3) for derivation formula.

Fig9. Relationship between obstacle and estimated trajectory

Fig10. o_i score as function of the interference with obstacle

o_i: Distance of trajectory without obstacle interference*. o_max: Total distance of trajectory.
*Obstacle interference to lerance up to 0.17[m].

Update frequency evaluation (E_f)

E_f evaluates the entire estimated trajectory. This is an index to evaluate the update frequency of position estimation. Evaluate whether position estimation is made in detail with respect to time.
Evaluation method: The score is calculated from the time difference from the estimated position to the update of the next estimated position (Fig.11). It is desirable that the update frequency is 1 [Hz] or more. Trajectory update frequency uses the equation (4). Refer to (5) for derivation formula.

u_t : Update trajectory of t seconds. u_t-1: Update trajectory of t-1 seconds.

Fig11. f_i score as function of the trajectory update frequency

f_i: Count when the trajectory update interval is 1[Hz] or more. f_max: Total time of trajectory.

Comprehensive evaluation (C.E.)

The comprehensive score is calculated based on the score of each evaluation indicator as shown in Fig.12. Refer to (6) for derivation formula.

Fig12. Percentage of each evaluation index in comprehensive evaluation

This evaluation will be the final score of the competition track 4.

Important Dates

: Open request for admission to competition

(real-admission, required to send technical description); From Feb.6th,2017(Mon) to July 15th,2017(Sat)
: Notification of admission; Shortly after requested by e-mail
: Submission of competitor's result; Before Sep.8th,2017(Fri)
: Submission of paper (option); 30 July 2017, AoE

Contact points and information

The contents in this page may be changed according to future discussion with the PDR benchmark standardization committee members, the participants of this competition, etc.
You can check change logs on this page.

Please contact us freely via pdr-warehouse2017@aist.go.jp

If you need a private contact, you can write to the track chairs:
・Masakatsu Kourogi (m.kourogi@aist.go.jp), AIST, Tsukuba Japan
・Ryosuke Ichikari (r.ichikari@aist.go.jp), AIST, Tsukuba Japan