We are glad to inform you that the 2019 AI City Challenge evaluation system is ready and open for submissions. The evaluation system can be found at https://eval.aicitychallenge.org. Please use the “Sign Up” link in the upper-right corner to sign up for an evaluation account. The system requires an email as the username and allows you to select your own password. We request that you use the same email in the evaluation system as one of the team member emails in your original dataset request. This email will be used to validate your request and provide you results submission access. After filling out your team details and submitting your evaluation account request, you will receive an email asking you to verify the account. Once you verify the account, you can log in to the evaluation system. However, you will not be able to submit results until an administrator approves your submission access. Please allow 24-48 hours for this approval to be completed.

When your account has been approved to submit results, the “Add Submission” button on the Submissions page will be enabled and you will be able to submit to any of the tracks. Please note that several limits have been imposed. Each team may only submit up to 5 results per track per day (Pacific time) and may only submit a total of 20 results for tracks 1 and 2 and 10 results for track 3. Note also that the results displayed in the submissions tables and on the leaderboard are computed on a 50% subset of the test data. After the competition deadline, the system will automatically display evaluation results based on the full test set. The leaderboard will only display the top 3 results and your current rank (if not in the top-3) before the competition deadline, but will show the entire list of best team submissions at the end of the competition, including team names (please choose a descriptive team name).