Submissions, SubmissionStatuses, SubmissionBundles¶

What are Submissions?¶

In Synapse, a Submission is your entry to a challenge or evaluation queue. When you participate in a computational challenge or collaborative project, you submit your work (such as predictions, models, or analysis results) as a Submission to be evaluated and scored.

Key Concepts¶

Before working with Submissions, it's helpful to understand how they fit into Synapse:

Entity: Your actual work stored in Synapse (like a File containing predictions or a Docker image with your model)
Evaluation: A queue that accepts and organizes submissions for a specific challenge or project
Submission: The object associated with submitting your Entity to an Evaluation queue, creating a record that can be tracked and scored

How Submissions Work¶

When you submit an Entity to an Evaluation: - The Submission creates an immutable record linking your Entity to that Evaluation - The Evaluation owns this Submission record (not you as the submitter) - Organizers can add scores and feedback through a SubmissionStatus object - You can track your Submissions and view their statuses

The Python client provides three object types for working with Submissions:

Submission: Represents your entry in an Evaluation queue
SubmissionStatus: Tracks scoring information and feedback for a Submission
SubmissionBundle: Combines a Submission and its SubmissionStatus for convenient access

What You'll Learn¶

This tutorial covers two perspectives:

Participating in challenges: Making and tracking your submissions
Organizing challenges: Scoring and managing submissions from participants

Tutorial Purpose¶

In this tutorial:

As a participant of a Synapse challenge, you will

Make a submission to an existing evaluation queue on Synapse
Fetch your existing submission
Count your submissions
Fetch all of your submissions from an existing evaluation queue on Synapse
Check the status of your submission
Cancel your submission

As an organizer of a Synapse challenge, you will

Annotate a submission to score it
Batch-update submission statuses
Fetch the submission bundle for a given submission
Allow cancellation of submissions
Delete submissions

Prerequisites¶

You have completed the Evaluation tutorial, or have an existing Evaluation on Synapse to work from
You have an existing entity with which to make a submission (can be a File or Docker Repository)
You have the correct permissions on the Evaluation queue for your desired tutorial section (participant or organizer)

1. Participating in a Synapse challenge¶

1. Make a submission to an existing evaluation queue on Synapse¶

# ==============================================================================
# 1. Make a submission to an existing evaluation queue on Synapse
# ==============================================================================

print("\n=== 1. Making a submission ===")

# Create a new Submission object
submission = Submission(
    entity_id=ENTITY_ID, evaluation_id=EVALUATION_ID, name="My Tutorial Submission"
)

# Submit the entity to the evaluation queue
submission = submission.store()

print(f"Submission created successfully!")
print(f"Submission ID: {submission.id}")
print(f"Submitted Entity: {submission.entity_id}")
print(f"Evaluation: {submission.evaluation_id}")
print(f"Submission Name: {submission.name}")
print(f"Created On: {submission.created_on}")

# Store the submission ID for later use
submission_id = submission.id

2. Fetch your existing submission¶

# ==============================================================================
# 2. Fetch your existing submission
# ==============================================================================

print("\n=== 2. Fetching existing submission ===")

# Retrieve the submission we just created
retrieved_submission = Submission(id=submission_id).get()

print(f"Retrieved submission:")
print(f"  ID: {retrieved_submission.id}")
print(f"  Name: {retrieved_submission.name}")
print(f"  Entity ID: {retrieved_submission.entity_id}")
print(f"  Submitter: {retrieved_submission.submitter_alias}")
print(f"  Created On: {retrieved_submission.created_on}")

3. Count your submissions¶

# ==============================================================================
# 3. Count your submissions
# ==============================================================================

print("\n=== 3. Counting submissions ===")

# Get the total count of submissions for this evaluation
submission_count = Submission.get_submission_count(evaluation_id=EVALUATION_ID)

print(f"Total submissions in evaluation: {submission_count}")

# Get count of submissions with specific status (optional)
scored_count = Submission.get_submission_count(
    evaluation_id=EVALUATION_ID, status="SCORED"
)

print(f"SCORED submissions in evaluation: {scored_count}")

4. Fetch all of your submissions from an existing evaluation queue on Synapse¶

# ==============================================================================
# 4. Fetch all of your submissions from an existing evaluation queue
# ==============================================================================

print("\n=== 4. Fetching all your submissions ===")

# Get all of your submissions for this evaluation
user_submissions = list(Submission.get_user_submissions(evaluation_id=EVALUATION_ID))

print(f"Found {len(user_submissions)} submissions from the current user:")
for i, sub in enumerate(user_submissions, 1):
    print(f"  {i}. ID: {sub.id}, Name: {sub.name}, Created: {sub.created_on}")

5. Check the status of your submission¶

# ==============================================================================
# 5. Check the status of your submission
# ==============================================================================

print("\n=== 5. Checking submission status ===")

# Fetch the status of our submission
status = SubmissionStatus(id=submission_id).get()

print(f"Submission status details:")
print(f"  Status: {status.status}")
print(f"  Modified On: {status.modified_on}")
print(f"  Can Cancel: {status.can_cancel}")
print(f"  Cancel Requested: {status.cancel_requested}")

# Check if there are any submission annotations (scores, feedback, etc.)
if status.submission_annotations:
    print(f"  Submission Annotations:")
    for key, value in status.submission_annotations.items():
        print(f"    {key}: {value}")
else:
    print(f"  No submission annotations available")

6. Cancel your submission¶

# ==============================================================================
# 6. Cancel your submission (optional)
# ==============================================================================

print("\n=== 6. Cancelling submission ===")

# Note: Only cancel if the submission allows it
# Uncomment the following lines if you want to test cancellation:

# cancelled_submission = submission.cancel()
# print(f"Submission {cancelled_submission.id} has been requested for cancellation")
#
# # Check the updated status
# updated_status = SubmissionStatus(id=submission_id).get()
# print(f"Cancel requested: {updated_status.cancel_requested}")

print(f"\nCancellation is commented out by default.")
print(f"Uncomment the cancellation code if you want to test this functionality.")

2. Organizing a Synapse challenge¶

1. Annotate a submission to score it¶

# ==============================================================================
# 1. Annotate a submission to score it
# ==============================================================================

print("\n=== 1. Annotating a submission with scores ===")

# First, get the submission status
status = SubmissionStatus(id=SUBMISSION_ID).get()
print(f"Retrieved submission status for submission {SUBMISSION_ID}")
print(f"Current status: {status.status}")

# Update the submission status with scoring information
status.status = "SCORED"
status.submission_annotations = {
    "accuracy": [0.85],
    "precision": [0.82],
    "feedback": ["Good performance!"],
    "validation_errors": "None detected",
    "score_errors": "None detected",
}

# Store the updated status
updated_status = status.store()
print(f"Successfully scored submission!")
print(f"Status: {updated_status.status}")
print(f"Annotations added:")
for key, value in updated_status.submission_annotations.items():
    print(f"  {key}: {value}")

2. Batch-update submission statuses¶

# ==============================================================================
# 2. Batch-update submission statuses
# ==============================================================================

print("\n=== 2. Batch updating submission statuses ===")

# First, get all submission statuses that need updating
statuses_to_update = SubmissionStatus.get_all_submission_statuses(
    evaluation_id=EVALUATION_ID,
    status="RECEIVED",  # Get submissions that haven't been scored yet
)

print(f"Found {len(statuses_to_update)} submissions to batch update")

if statuses_to_update:
    # Update each status with validation information
    for i, status in enumerate(statuses_to_update):
        status.status = "VALIDATED"
        status.submission_annotations = {
            "validation_status": ["PASSED"],
            "validation_timestamp": ["2024-11-24T10:30:00Z"],
            "batch_number": [i + 1],
            "validator": ["automated_system"],
        }

    # Perform batch update
    batch_response = SubmissionStatus.batch_update_submission_statuses(
        evaluation_id=EVALUATION_ID,
        statuses=statuses_to_update,
        is_first_batch=True,
        is_last_batch=True,
    )

    print(f"Batch update completed successfully!")
    print(f"Batch response: {batch_response}")
else:
    print("No submissions found with 'RECEIVED' status to update")

3. Fetch the submission bundle for a given submission¶

# 3. Fetch the submission bundle for a given submission
# ==============================================================================

print("\n=== 3. Fetching submission bundle ===")

# Get all submission bundles for the evaluation
print("Fetching all submission bundles for the evaluation...")

bundles = list(
    SubmissionBundle.get_evaluation_submission_bundles(
        evaluation_id=EVALUATION_ID, status="SCORED"  # Only get scored submissions
    )
)

print(f"Found {len(bundles)} scored submission bundles")

for i, bundle in enumerate(bundles[:5]):  # Show first 5
    submission = bundle.submission
    status = bundle.submission_status

    print(f"\nBundle {i + 1}:")
    if submission:
        print(f"  Submission ID: {submission.id}")
        print(f"  Submitter: {submission.submitter_alias}")
        print(f"  Entity ID: {submission.entity_id}")
        print(f"  Created: {submission.created_on}")

    if status:
        print(f"  Status: {status.status}")
        print(f"  Modified: {status.modified_on}")
        if status.submission_annotations:
            print(f"  Scores:")
            for key, value in status.submission_annotations.items():
                if key in ["accuracy", "f1_score", "precision", "recall"]:
                    print(f"    {key}: {value}")

4. Allow cancellation of submissions¶

# 4. Allow cancellation of submissions
# ==============================================================================

print("\n=== 4. Managing submission cancellation ===")

# First, check if any submissions have requested cancellation
all_statuses = SubmissionStatus.get_all_submission_statuses(evaluation_id=EVALUATION_ID)

cancellation_requests = [status for status in all_statuses if status.cancel_requested]

print(f"Found {len(cancellation_requests)} submissions with cancellation requests")

# Process cancellation requests
for status in cancellation_requests:
    print(f"Processing cancellation request for submission {status.id}")

    # Update to allow cancellation (organizer decision)
    status.can_cancel = True
    status.status = "CLOSED"
    status.submission_annotations.update(
        {
            "cancellation_reason": ["User requested cancellation"],
            "cancelled_by": ["organizer"],
            "cancellation_date": ["2024-11-24"],
        }
    )

    # Store the update
    updated_status = status.store()
    print(f"  Approved cancellation for submission {updated_status.id}")

# Example: Proactively allow cancellation for a specific submission
print("\nEnabling cancellation for a specific submission...")
target_status = SubmissionStatus(id=SUBMISSION_ID).get()
target_status.can_cancel = True
target_status = target_status.store()
print(f"Cancellation enabled for submission {SUBMISSION_ID}")

# ==============================================================================
# 5. Delete submissions

5. Delete submissions¶

# print("\n=== 5. Deleting submissions ===")
# print("Finding and deleting submissions that have been requested for cancellation...")

# # Get all submission statuses to check for cancellation requests
# all_statuses = SubmissionStatus.get_all_submission_statuses(
#     evaluation_id=EVALUATION_ID,
# )

# # Find submissions that have been requested for cancellation
# submissions_to_delete = []
# for status in all_statuses:
#     if status.cancel_requested:
#         submissions_to_delete.append(status.id)

# print(f"Found {len(submissions_to_delete)} submissions with cancellation requests")

# # Delete each submission that was requested for cancellation
# for submission_id in submissions_to_delete:
#     submission = Submission(id=submission_id).get()
#     submission.delete()
#     print(f"Successfully deleted submission {submission_id}")

# if submissions_to_delete:
#     print(f"Completed deletion of {len(submissions_to_delete)} requested submissions")

print(f"\nDeletion step is commented out by default.")
print(f"Uncomment the deletion code if you want to test this functionality.")

print(f"\n=== Organizer tutorial completed! ===")

Source code for this tutorial¶

Click to show me (source code for Participant)

"""
Submission Participant Tutorial - Code for working with Submissions as a challenge participant.

This tutorial demonstrates how to:
1. Make a submission to an existing evaluation queue
2. Fetch your existing submission
3. Count your submissions
4. Fetch all of your submissions from an evaluation queue
5. Check the status of your submission
6. Cancel your submission
"""

from synapseclient import Synapse
from synapseclient.models import Submission, SubmissionStatus

syn = Synapse()
syn.login()

# REQUIRED: Set these to your actual Synapse IDs
# Do NOT leave these as None - the script will not work properly
EVALUATION_ID = None  # Replace with the evaluation queue ID you want to submit to
ENTITY_ID = None  # Replace with the entity ID you want to submit

assert EVALUATION_ID is not None, "EVALUATION_ID must be set to the evaluation queue ID"
assert (
    ENTITY_ID is not None
), "ENTITY_ID must be set to the entity ID you want to submit"

print(f"Working with Evaluation: {EVALUATION_ID}")
print(f"Submitting Entity: {ENTITY_ID}")

# ==============================================================================
# 1. Make a submission to an existing evaluation queue on Synapse
# ==============================================================================

print("\n=== 1. Making a submission ===")

# Create a new Submission object
submission = Submission(
    entity_id=ENTITY_ID, evaluation_id=EVALUATION_ID, name="My Tutorial Submission"
)

# Submit the entity to the evaluation queue
submission = submission.store()

print(f"Submission created successfully!")
print(f"Submission ID: {submission.id}")
print(f"Submitted Entity: {submission.entity_id}")
print(f"Evaluation: {submission.evaluation_id}")
print(f"Submission Name: {submission.name}")
print(f"Created On: {submission.created_on}")

# Store the submission ID for later use
submission_id = submission.id

# ==============================================================================
# 2. Fetch your existing submission
# ==============================================================================

print("\n=== 2. Fetching existing submission ===")

# Retrieve the submission we just created
retrieved_submission = Submission(id=submission_id).get()

print(f"Retrieved submission:")
print(f"  ID: {retrieved_submission.id}")
print(f"  Name: {retrieved_submission.name}")
print(f"  Entity ID: {retrieved_submission.entity_id}")
print(f"  Submitter: {retrieved_submission.submitter_alias}")
print(f"  Created On: {retrieved_submission.created_on}")

# ==============================================================================
# 3. Count your submissions
# ==============================================================================

print("\n=== 3. Counting submissions ===")

# Get the total count of submissions for this evaluation
submission_count = Submission.get_submission_count(evaluation_id=EVALUATION_ID)

print(f"Total submissions in evaluation: {submission_count}")

# Get count of submissions with specific status (optional)
scored_count = Submission.get_submission_count(
    evaluation_id=EVALUATION_ID, status="SCORED"
)

print(f"SCORED submissions in evaluation: {scored_count}")

# ==============================================================================
# 4. Fetch all of your submissions from an existing evaluation queue
# ==============================================================================

print("\n=== 4. Fetching all your submissions ===")

# Get all of your submissions for this evaluation
user_submissions = list(Submission.get_user_submissions(evaluation_id=EVALUATION_ID))

print(f"Found {len(user_submissions)} submissions from the current user:")
for i, sub in enumerate(user_submissions, 1):
    print(f"  {i}. ID: {sub.id}, Name: {sub.name}, Created: {sub.created_on}")

# ==============================================================================
# 5. Check the status of your submission
# ==============================================================================

print("\n=== 5. Checking submission status ===")

# Fetch the status of our submission
status = SubmissionStatus(id=submission_id).get()

print(f"Submission status details:")
print(f"  Status: {status.status}")
print(f"  Modified On: {status.modified_on}")
print(f"  Can Cancel: {status.can_cancel}")
print(f"  Cancel Requested: {status.cancel_requested}")

# Check if there are any submission annotations (scores, feedback, etc.)
if status.submission_annotations:
    print(f"  Submission Annotations:")
    for key, value in status.submission_annotations.items():
        print(f"    {key}: {value}")
else:
    print(f"  No submission annotations available")

# ==============================================================================
# 6. Cancel your submission (optional)
# ==============================================================================

print("\n=== 6. Cancelling submission ===")

# Note: Only cancel if the submission allows it
# Uncomment the following lines if you want to test cancellation:

# cancelled_submission = submission.cancel()
# print(f"Submission {cancelled_submission.id} has been requested for cancellation")
#
# # Check the updated status
# updated_status = SubmissionStatus(id=submission_id).get()
# print(f"Cancel requested: {updated_status.cancel_requested}")

print(f"\nCancellation is commented out by default.")
print(f"Uncomment the cancellation code if you want to test this functionality.")

print(f"\n=== Tutorial completed! ===")
print(f"Your submission ID {submission_id} is ready for evaluation.")
print(f"Check back later to see if the organizers have scored your submission.")

Click to show me (source code for Organizer)

"""
Submission Organizer Tutorial - Code for working with Submissions as a challenge organizer.

This tutorial demonstrates how to:
1. Annotate a submission to score it
2. Batch-update submission statuses
3. Fetch the submission bundle for a given submission
4. Allow cancellation of submissions
5. Delete submissions
"""

from synapseclient import Synapse
from synapseclient.models import Submission, SubmissionBundle, SubmissionStatus

syn = Synapse()
syn.login()

# REQUIRED: Set these to your actual Synapse IDs
# Do NOT leave these as None - the script will not work properly
EVALUATION_ID = None  # Replace with the evaluation queue ID you manage
SUBMISSION_ID = None  # Replace with a submission ID from your evaluation

assert (
    EVALUATION_ID is not None
), "EVALUATION_ID must be set to the evaluation queue ID you manage"
assert (
    SUBMISSION_ID is not None
), "SUBMISSION_ID must be set to a submission ID from your evaluation"

print(f"Working with Evaluation: {EVALUATION_ID}")
print(f"Managing Submission: {SUBMISSION_ID}")

# ==============================================================================
# 1. Annotate a submission to score it
# ==============================================================================

print("\n=== 1. Annotating a submission with scores ===")

# First, get the submission status
status = SubmissionStatus(id=SUBMISSION_ID).get()
print(f"Retrieved submission status for submission {SUBMISSION_ID}")
print(f"Current status: {status.status}")

# Update the submission status with scoring information
status.status = "SCORED"
status.submission_annotations = {
    "accuracy": [0.85],
    "precision": [0.82],
    "feedback": ["Good performance!"],
    "validation_errors": "None detected",
    "score_errors": "None detected",
}

# Store the updated status
updated_status = status.store()
print(f"Successfully scored submission!")
print(f"Status: {updated_status.status}")
print(f"Annotations added:")
for key, value in updated_status.submission_annotations.items():
    print(f"  {key}: {value}")

# ==============================================================================
# 2. Batch-update submission statuses
# ==============================================================================

print("\n=== 2. Batch updating submission statuses ===")

# First, get all submission statuses that need updating
statuses_to_update = SubmissionStatus.get_all_submission_statuses(
    evaluation_id=EVALUATION_ID,
    status="RECEIVED",  # Get submissions that haven't been scored yet
)

print(f"Found {len(statuses_to_update)} submissions to batch update")

if statuses_to_update:
    # Update each status with validation information
    for i, status in enumerate(statuses_to_update):
        status.status = "VALIDATED"
        status.submission_annotations = {
            "validation_status": ["PASSED"],
            "validation_timestamp": ["2024-11-24T10:30:00Z"],
            "batch_number": [i + 1],
            "validator": ["automated_system"],
        }

    # Perform batch update
    batch_response = SubmissionStatus.batch_update_submission_statuses(
        evaluation_id=EVALUATION_ID,
        statuses=statuses_to_update,
        is_first_batch=True,
        is_last_batch=True,
    )

    print(f"Batch update completed successfully!")
    print(f"Batch response: {batch_response}")
else:
    print("No submissions found with 'RECEIVED' status to update")

# ==============================================================================
# 3. Fetch the submission bundle for a given submission
# ==============================================================================

print("\n=== 3. Fetching submission bundle ===")

# Get all submission bundles for the evaluation
print("Fetching all submission bundles for the evaluation...")

bundles = list(
    SubmissionBundle.get_evaluation_submission_bundles(
        evaluation_id=EVALUATION_ID, status="SCORED"  # Only get scored submissions
    )
)

print(f"Found {len(bundles)} scored submission bundles")

for i, bundle in enumerate(bundles[:5]):  # Show first 5
    submission = bundle.submission
    status = bundle.submission_status

    print(f"\nBundle {i + 1}:")
    if submission:
        print(f"  Submission ID: {submission.id}")
        print(f"  Submitter: {submission.submitter_alias}")
        print(f"  Entity ID: {submission.entity_id}")
        print(f"  Created: {submission.created_on}")

    if status:
        print(f"  Status: {status.status}")
        print(f"  Modified: {status.modified_on}")
        if status.submission_annotations:
            print(f"  Scores:")
            for key, value in status.submission_annotations.items():
                if key in ["accuracy", "f1_score", "precision", "recall"]:
                    print(f"    {key}: {value}")

# ==============================================================================
# 4. Allow cancellation of submissions
# ==============================================================================

print("\n=== 4. Managing submission cancellation ===")

# First, check if any submissions have requested cancellation
all_statuses = SubmissionStatus.get_all_submission_statuses(evaluation_id=EVALUATION_ID)

cancellation_requests = [status for status in all_statuses if status.cancel_requested]

print(f"Found {len(cancellation_requests)} submissions with cancellation requests")

# Process cancellation requests
for status in cancellation_requests:
    print(f"Processing cancellation request for submission {status.id}")

    # Update to allow cancellation (organizer decision)
    status.can_cancel = True
    status.status = "CLOSED"
    status.submission_annotations.update(
        {
            "cancellation_reason": ["User requested cancellation"],
            "cancelled_by": ["organizer"],
            "cancellation_date": ["2024-11-24"],
        }
    )

    # Store the update
    updated_status = status.store()
    print(f"  Approved cancellation for submission {updated_status.id}")

# Example: Proactively allow cancellation for a specific submission
print("\nEnabling cancellation for a specific submission...")
target_status = SubmissionStatus(id=SUBMISSION_ID).get()
target_status.can_cancel = True
target_status = target_status.store()
print(f"Cancellation enabled for submission {SUBMISSION_ID}")

# ==============================================================================
# 5. Delete submissions
# ==============================================================================

# print("\n=== 5. Deleting submissions ===")
# print("Finding and deleting submissions that have been requested for cancellation...")

# # Get all submission statuses to check for cancellation requests
# all_statuses = SubmissionStatus.get_all_submission_statuses(
#     evaluation_id=EVALUATION_ID,
# )

# # Find submissions that have been requested for cancellation
# submissions_to_delete = []
# for status in all_statuses:
#     if status.cancel_requested:
#         submissions_to_delete.append(status.id)

# print(f"Found {len(submissions_to_delete)} submissions with cancellation requests")

# # Delete each submission that was requested for cancellation
# for submission_id in submissions_to_delete:
#     submission = Submission(id=submission_id).get()
#     submission.delete()
#     print(f"Successfully deleted submission {submission_id}")

# if submissions_to_delete:
#     print(f"Completed deletion of {len(submissions_to_delete)} requested submissions")

print(f"\nDeletion step is commented out by default.")
print(f"Uncomment the deletion code if you want to test this functionality.")

print(f"\n=== Organizer tutorial completed! ===")