ArcGIS Collector Illegal Start of Token

After minutes, hours, nay - week in the field, you decide it’s time to sync your offline ArcGIS Collector edits so others in your organization can enjoy the vast quantities of field data you recently collected. Clicking the sync button, a familiar progress bar appears and we start thinking about our next Collector excursion and how easy that was. Alas, we jinxed it. Our positive, happy thoughts somehow caused a rift in the wifi continuum. It then appears: ArcGIS Collector Illegal Start of Token [<].

Sync Error: ArcGIS Collector Illegal Start of Token

You try syncing many times thinking the gremlins probably went away, but the consistency of the message makes you second guess you and your finger mashing of the sync button. You think possibly checking for an update in the App store could help - but hopefully you finish reading this before doing so, or risk losing all that glorious data.

ArcGIS Collector Illegal Start of Token [>]:

The illegal start of token message is very generic, and there are many different reasons you might get this message. The one described below is based on the following configuration:

Apple iPad Air 2, iOS 9.02
ArcGIS Collector v10.3.4 or v10.3.5
Service locally hosted on ArcGIS Server 10.3.1 (windows/IIS),
- Secured service
- Service has 1 line FC, 1 point FC, 1 photo attachment to point FC
Service is registered with ArcGIS Online (AGOL) as Feature Service
The Feature Service SAVES service credentials
Create AGOL map referencing the feature service
Easy peasy

The real culprit in our situation is the credentials being stored with the feature service when added to ArcGIS Online (AGOL). For some reason, the sync will start but there is a failure after a few minutes with the token. Well, it’s actually the combination of the stored credentials, and having a large sync process - not based on feature count, but total size (photos included in that size consideration). Photos being the main focus of contention here as they drastically increase the total size.

Solutions - Part 1: Worth Trying First

Since the error is generic, it’s a good idea to start with the simple solutions that appear to have worked for a few users. Although these didn’t work in our situation, GeoNet has many possible solutions that should be tested first:

Increasing the IIS uploadReadAheadSize on our ArcGIS Server web server
Using a fast and strong wifi connection
Wait until ~~10.3.5~~ errr, 10.3.6 when the app will allow photo resize to deal with total sync size and hopefully fix the sync bug regardless
Whatever you do - don’t upgrade the app from 10.3.4 to 10.3.5 thinking it will fix the issue. This introduces another bug: Upgrading from 10.3.4 to 10.3.5 will erase all offline edits (found out the hard way).

Solutions - Part 2: Deep Dive

If part 1 above doesn’t solve the sync issue, there are a few additional methods that may fix or at least save the synced edits as we wait for 10.3.6.

1: Recreate the Feature Service: This time don’t save the credentials with the user/pw. Then open the existing AGOL map, remove the old Feature Service, and add the new one. This will NOT fix the sync of existing offline edits - take care of those first or extract manually at least and try the second step below. The sync of existing offline edits will fail since the Feature Service has a new reference ID in AGOL - In a future release of AGOL, hopefully we can modify the existing service to prompt no longer storing credentials - this might allow existing edits to be saved with less effort.

With the new approach users will be prompted for the user/pw when first accessing the Collector Map with the Feature Service inside. This isn’t ideal, but it does save this information in Collector, so as long as the Collector account stays logged in, the service seems to keep the feature service user/pw stored. You can also store the user/pw in the Notes app just in case (we won’t tell).

This approach has allowed us to sync larger batches of data, however, we sync every night as we still don’t know if there is a cap on the effectiveness. Very large sets of attachments should still consider a different approach (online edits, or store Feature Service on AGOL) as there might still be a limit.

2: Manually sync edits: Yuck! If nothing else works buckle down and get ready for traditional GIS joins and calculates. Since we used an iPad, the first thing to do is get the data off the device. Yes, we need iTunes (sorry). With iTunes connected to the iPad, you can copy the sync edits by browsing Apps Data section. Just copy the user folder with edits to the desktop. The files inside have names like “3123456789.geodatabase” but don’t worry, they are legitimate GDBs.

iTunes Collector Documents: Get your data back!

Since this is a Geodatabase in disguise, we first need to get the data into a usable format. Thankfully, there are now tools to get the data into an XML workspace. To do this, just open the Toolbox or use python to extract the data: arcpy.ExportXMLWorkspaceDocument_management(source, XML). Once complete, you can import the XML into a new/empty File GDB using another Toolbox/python tool arcpy.ImportXMLWorkspaceDocument_management(XML, GDB). Congratulations, you now have a File Geodatabase with Collector edits.

This approach will keep the relationship and GUIDs intact. If there are no relationships or photo attachments, and only new features created - just use the Object Loader (ArcMap) or Load Data (ArcCatalog) tools to append the new data into your Enterprise Geodatabase.

If there are attachments/photo and possible modifications to existing features, you still have your work cut-out for you. The best approach I found so far is to add an additional “Original_GUID” field to all Feature Classes and Tables (in source/enterprise GDB, and in newly created file GDB). Calculate the File GDB “Original_GUID” field equal to the real GUID field in Feature Classes. In related tables, create a “Original_RelGUID” and calculate based on “Related_GUID”. Once complete on all tables and feature classes, you can start the old-school grind of loading, joining, and calculating fields.

For attachment tables, the “Related_GUID” field will not change when appended to a parent layer (so why did we create a copy?! You’ll see…). In the Feature Class, the “GUID” field will change when loaded, but the “Original_GUID” we populated will remain. You can now join the “Original_GUID” in FC to the “Original_RelGUID” in the related table. Next, calculate the Related_GUID in the table equal to the FC’s GUID field. Since we joined using the “Original_RelGUID”, the table will not go crazy since we aren’t changing the field the ArcMap Join is based on - otherwise, the table will lose track of the join and edits might not look like they have been made correctly. Also good for double checking your work (which is why we created it).