PDR Challenge in Warehouse Picking ロゴ

Track 4: "PDR for warehouse picking (off-site)"

News

2017/7/7
Updated information display frame added.
Added description of Competition data to "Description of Datasets".
2017/6/14
Added ability to move to top with Back to top.
2017/5/24
Added wearing image to "Phone holding".
Fixed explanation of "Phone holding".
Annotation added to "Sensor data of terminal".

Challenge Goal

The goal of this challenge is the precision improvement of positioning technique, sharing a common recognition of problems and establishment of a method to compare the warehouse technique for the logistics by competing the precision of the pedestrian dead reckoning(PDR) techniques using the picking data measured in a real warehouse for the logistics. Comparing to the usual walking, PDR positioning using the measuring data in a warehouse includes a variety of tasks like picking, gathering, walking in a warehouse, carrying goods with carts and so on. Therefore, the advanced PDR positioning techniques and its correction techniques to treat a variety of situations as mentioned above are needed.

Prize

The winner of Track 4 will be awarded one of the following prizes by PDR benchmark standardization committee (JP) according to the winner's preference.

1) PDR module (Sugihara SEI Co. &Lt., 200,000 JPY worth) + 100,000 JPY
PDR module

or

2) 150,000 JPY

Back to top

Main feature of the challenge

Off-site challenge approach 

This challenge is done off-site, so all data processing will be done using sensor data measured by 8 people in a warehouse for about 3 hours for each. For evaluation standard, competitors need to use the WMS picking logs recorded during measuring the sensor data.

Multiple sources of information

As the sensor data for PDR, the gyro data, the acceleration data, the magnetic data, the pressure data and BLE beacon data were measured using the android smartphones (LG Nexus5). The smartphone was mounted by the warehouse worker and the wearing position was kept as same as possible.

Continuous motion and recording process

The test subjects of this challenge were the workers picking goods in a warehouse. The expected actions of the picking workers are as follows.
  • Walking in a warehouse
  • Vouching
  • Picking at shelves (including the operating the terminals)
  • Carrying goods using carts
Information about correct actions of each time are not given.

Realistic walking style

As mentioned above, during data measurement, the test subjects do not always walk. They do several types of behaviors while changing their position. The selection of the moving route for the picking instruction was left up to the workers in a warehouse, so while walking a variety of moving ways are possible like stop on the route, change the direction, side stepping and rear walking.

Phone holding

The smartphone is fixed to the waist with a waist pouch as shown in the figure.

スマホ装着図

Back to top

Desired localization approaches

By the nature of the data measured in this challenge, the positioning methods used by the contest participants are desired to have following abilities.
  • PDR method which has robust positioning functions for moving in a warehouse
  • Functions to cancel out accumulated errors by given discrete positional refeference and RSSI of BLE beacons
  • Time series optimization using the given data
  • Keeping naturality of attitude during the picking action (picking position, amount of moving)
  • Keeping naturality of walking speed
  • Function to avoid incursion into the area of objections

Warehouse

Logistics warehouse A

Logistics warehouse A

Click the image to see the enlarged image

Logistics warehouse B

Logistics warehouse B

Click the image to see the enlarged image

Back to top

Description of Datasets

Description of Competition data

Warehouse A is sample data showing long-term measurement in the logistics warehouse.
Warehouse B is data to be used in the real competition, and sample data of Warehouse B extra only 30 minutes from the measurement start time.

The time (in Unixtime) of the range of the measurement in Warehouse B of this competition is as follows.

Terminal_ID,Measurement_start_time,Wearing_start_time,Measurement_end_time

1,1485239145,1485240508,1485252057
2,1485239148,1485240513,1485248722
3,1485239234,1485240524,1485252460
4,1485239243,1485240545,1485252313
5,1485239286,1485240550,1485252407
6,1485239291,1485240555,1485252344
7,1485239328,1485240562,1485252227
8,1485239334,1485240568,1485252085

"Terminal_ID" is a number for distinguishing terminals. "Measurement_start_time" is the time when the terminal started recording sensor data. "Wearing_start_time" is the time when the terminal was actually worn on the employee's body. Before the workers worn the terminals, the terminals were arranged on a table. "Measurement_end_time" is the time when the terminal stop recording at the measurement.

Actual evaluation points for real competition are existed between wearing_start_time and measurement_end_time.

We are disclosing the data which is extracted only first 30 minutes of warehouse B as sample data. Data extraction was performed on "Sensor data of terminal(Working data)" and "WMS". Note that sample WMS data include WMS logs before Measurement_start_time, you can ignore the data outside of the range of measurement.

Back to top

Dataset list for logistics warehouse A

Sensor data of terminal

Note: The data format provided is what we defined, but since smartphones are used for measurements this time, the unmeasured sensor data is included in our data format. Specifically, the values of gyro temperature (xyz) and air temperature are written as '0' , but they are ignorable.
Sensor data when the terminal is placed on a stationary flat surface (.txt)
Format
Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
Sensor data when writing a figure 8 in the air with the terminal in hand (.txt)
Format
Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
Sensor data when picking with the terminal attached (.txt)
Format
Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t

Map information

Map image (.png)
Map size (.csv)
Format
The coordinates of the four corners of the map and the coordinates at the start (xyz)
Shelf arrangement (.csv)
Format
Shelf number, Arrangement coordinates (xyz), Width, Depth, Direction to pick

WMS data

Picking operation (.csv)
Format
Worker number, Time picked, Shelf number, Number of picks

Back to top

Dataset list for logistics warehouse B

Sensor data of terminal

Sensor data when the terminal is placed on a stationary flat surface (.txt)
Format
Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
Sensor data when writing a figure 8 in the air with the terminal in hand (.txt)
Format
Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t
Sensor data when picking with the terminal attached (.txt)
Format
Unixtime, Angular velocity (xyz), Acceleration (xyz), Magnetism (xyz), Gyro temperature(xyz), Temperature, Atmospheric pressure, ⊿t, BLE reception information

Map information

Map image (.png)
Map size (.csv)
Format
The coordinates of the four corners of the map and the coordinates at the start (xyz)
Shelf arrangement (.csv)
Format
Shelf number, Arrangement coordinates (xyz), Width, Depth, Direction to pick

WMS data

Picking operation (.csv)
Format
Worker number, Time picked, Shelf number, Number of picks

BLE beacon information

BLE beacon information
Format
MAC address, Installation coordinates (xyz)

Back to top

How to participate?

Competitors are requested to follow the steps as mentioned below for participating this challenge.

Step1 Request for admission

During the registration period, contestant of the competitions can register themselves as the competitors of the PDR challenge.

 For this track 4, admission process can be separated into pre-admission, and real-admission.
If you have an interest for joining PDR challenge, please send an e-mail to pdr-warehouse2017@aist.go.jp with your self-introduction (at least with your name and institution). Then we will provide the access pass (ID and Password) for downloading sample data. We regard this e-mail as your pre-admission for track 4.
Real-registration follows the same Admission Process with the other tracks.

 

[Admission Process]
A "competitor" can be any individual or group of individuals working as a single team, associated to a single or a number of organizations, who wants to participate in one or several tracks.

 

Competitors apply for admission to the competition tracks by providing a short (2 to 4 pages) "technical description" of their localization system, including a description of the algorithms and protocols used. The technical description must be sent by e-mail to the chairs of the intended track. Track chairs will accept or refuse the application in a short time, based on technical feasibility and logistic constraints.

 

After acceptance of the competitor's technical description, one member of the competing team is required to register to the IPIN conference, specifying that the registration is linked to a competition track. Full registration to the conference covers participation in the competition process (allocated time, support and space) and the submission of a paper describing the system.

 

Competitors are required to make an oral presentation of their system during a dedicated session at the IPIN conference. Additionally, competitor teams are invited, but not required, to submit a paper to the conference. The paper will follow the same peer review and publishing process as all the other papers submitted to IPIN. If the paper is work-in-progress, competitors can benefit from relaxed deadline as specified below.

 

Competitors who wish to test their system, but do not want to be included in the public rankings and compared with other competitors, can ask for that special condition in advance to the chairs of the relevant track.

Step2 Downloading sample data

 If the pre-admission for the PDR challenge is completed, competitors can obtain a password and a user ID for downloading the sample data of the PDR challenge so that they start preparing the competition.

Step3 Registration of IPIN conference

 If you decide to join the track 4, you are required to complete real-admission. In order to complete real-admission, you need to submit a short (2 to 4 pages) "technical description" of their localization system, including a description of the algorithms and protocols used. And you (at least one member from your group) are also required to register to the IPIN conference.
Sample data provided for whom they completed pre-admission are not the final data for the competition. Actual test data and participation rights for the competition are exclusively provided for competitors who completed also completed the conference registration of the IPIN2017.

Step4 Access to the test data for final evaluation

Actual data for the competition are provided after confirmation of the completion of the conference registration. The competitors can obtain another user ID and a password for downloading the actual data for the competition.

Step5 Result submission

Competitors should submit their result of estimation of the trajectory with predetermined format which includes x,y coordinate of the target positions and orientation of the targets with timestamps. Definition of exact format and sample data for submission will be included in the actual data package described above.
Dead-line for result submission: 8 September 2017

Step6 Oral presentation of results at IPIN Conference

Competitors are required to present their methods and algorithms, and state their advantages of the methods and algorithms in the light of the evaluation criteria in the special session of the competitions during the conference. Additionally, competitors are invited, but not required, to submit a paper to the conference. At the end of the session, the competition organizers will show competitors' results and estimated trajectories. The results will be scored by a composite indicator according to the evaluation criteria.

Back to top

Evaluation metrics

The final metric will be based on:
  • The accuracy of the position estimation
  • The naturalness of estimated trajectories considering human's attitude
  • The unique criteria for tracking the picking workers

Details

The final score will be calculated by weighted sum of following individual scores.
The final comprehensive evaluation (C.E.) will be calculated as follows:
C.E=(Ed *0.2)+(Es *0.2)+(Ev *0.15)+(Ef *0.1)+(Ep *0.05)+(Eo *0.3)

The accuracy of the position estimation

Ed :
Ed is a score based on the Euclidian distance between the ground truth positions and the estimated positions.
Es:
In this challenge, the competitors can utilize given reference of picking data for cancelling the accumulated errors. Therefore accumulated error is also evaluated by separating the whole trajectory into sections subdivided by the picking data.
Es is a score based on maximum Euclidian error in each section.

The naturalness of estimated trajectories considering human's attitude

Ev:
The naturalness of walking speed will be scored. If the temporal speed exceed the threshold defined by average walking speed of human, The score Ev will be deducted by the exceeded speeds and the number of times of the excess. (point-deduction scoring)
Ef:
The frequency of position estimation will be scored.We determined minimal frequency of the estimation as 1 Hz. If the submitted trajectory include blanks (absences of the estimation during the period) of the estimation last for more than 1 sec, the score Ef will be deducted according to the length of the blanks and the number of the blanks.

The unique criteria for localizing the picking workers

Ep:
The correctness of attitude during the picking operation will be evaluated. In particular, workers are supposed to be stopped at the moment of picking operations. The walking speeds at the moments of the picking operation will be scored by the Ep.
Eo :
The collision between the estimated trajectories and the objections in the warehouse will be evaluated. In the warehouse, there are shelves as the objections. The score Eo will be deducted, if the estimated trajectories intrude into the area of the objections.

Back to top

Important Dates

Open request for admission to competition

(real-admission, required to send technical description)

From Feb.6th,2017(Mon) to July 15th,2017(Sat)
Notification of admission
Shortly after requested by e-mail
Submission of competitor's result
Before Sep.8th,2017(Fri)
Submission of paper (option)

30 July 2017, AoE

Contact points and information

The contents in this page may be changed according to future discussion with the PDR benchmark standardization committee members, the participants of this competition, etc.
You can check change logs on this page.
Please contact us freely via pdr-warehouse2017@aist.go.jp
If you need a private contact, you can write to the track chairs:
・Masakatsu Kourogi (m.kourogi@aist.go.jp), AIST, Tsukuba Japan
・Ryosuke Ichikari (r.ichikari@aist.go.jp), AIST, Tsukuba Japan

Back to top

Sponsors

multisoup

Supporting Communities

PDR Benchmark Standardization Committee

Previous PDR Challenge