python, developer, arcgis enterprise, arcgis server

Secure Map Service to Shapefile with Python Part 3

This post is inspired by the many emails I’ve received on the first two posts in this series. This article will address the most most popular question so far: How to work with a secure map service to Shapefile. Short answer, tokens!

Before We Begin

We aren’t covering Part 1 or Part 2 again, but we will continue to build on those concepts. Secure services use the same approach as before with one additional parameter that passes a security token with each request. So once we have the token, the original posts can be used for the rest of the plumbing.

What Are You Token About?

When ArcGIS web services are secured using ArcGIS token-based authentication, every request must be accompanied by a valid token. A token can be acquired in one of two ways:

Using the ArcGIS Server token request page: This method is very straight forward and requires very little change to your Python script. Just visit the token request page (http://gisserver.domain.com:6080/arcgis/tokens), add your details, and generate a token. I know, not very Pythonic. Even if you decide to use the programmatic approach, this page is very useful for testing and verifying the username password is valid.

Programmatically: Send an HTTP(s) request to the token endpoint, and receive the token back inside your code/script (Python in our case). But there are some limitations to this method - the username and password can’t be passed as query parameters directly in the URL as we do in every example in Part 1 and Part 2 with GET requests. Instead, parameters are passed in the body of a POST request. It is also recommended to use HTTPS when sending this information whenever possible.

Once the token is received using an above method, we can modify the parameters to include the token in each request.

Programmatically Request a Token From ArcGIS Server

To authenticate with ArcGIS Server we need to submit a name and password that has permissions to the service/folder. The server will return a token which must be include with all subsequent service requests we make to the server.

The token does not last forever; it is designed to time out so it cannot be stolen and used indefinitely by a malicious user. You have to request a new token each time you run your script (but not each time you make a request). ~ Esri Server Help

This relatively simple task does unfortunately require more lines of code (and modules) than one would expect when using Python 2x. There are lots of existing resources out there if you want to use urllib with either urllib2 or httplib modules to perform this task. However for this example we are going to use the requests module (manual pip install is required). Why use this external module? It only needs a few lines of code, and with a tag line of “Python HTTP for Humans”, how could you not want to use it?! To install on Windows 10, just open PowerShell and enter: c:\Python27\python.exe -m pip install requests

Once installed, requesting the token is pretty straight forward…

With this token now saved as the variable myToken, we can make service requests until the token expires. Since we didn’t specify the time in our request, it will be whatever default is set by ArcGIS Server (typical is 60 minutes). Now whenever making a queries to map services, just add an extra parameter to the service requests used in the previous articles. The order of the parameters is up to you.

Original request without token: https://domain/arcgis/rest/services/Public/Directory/MapServer/0/query?where=...
New request with token added: https://domain/arcgis/rest/services/Public/Directory/MapServer/0/query?token=myToken&where=...

The new requests will be validated by ArcGIS Server and we can continue to download the information as we did with open services. As a side note, this example of generating a token uses the ‘Requesting IP’. This also means the generated token can’t be shared on multiple computers. As well, if there is a proxy server between the client and ArcGIS Server, tokens generated using the IP Address and IP address of this request’s origin options must be bound to the IP address of the proxy server. If there are multiple load balancing proxy servers between the client and ArcGIS Server, the HTTP Referer option should be used [ArcGIS Server Help].

Secure Map Service to Shapefile with Python

That’s really all we need to access a secure map service. A valid token was the only missing piece of the puzzle. Although if trying to access AGOL, it’s easier to sign in, and download via the GUI.

Not all ArcGIS Server instances are configured the same unfortunately. There are some ArcGIS Server installs where the Generate Token request is unavailable. This is due to security concerns (HTTP vs HTTPS), or the lack of requirement to make it available. Some may also point out that it is possible to request a token via HTTP using GET. This is technically possible, although thankfully not enabled by default - since using a GET request may seem easier but credentials are provided as part of the URL and may be stored in browser history or in network components. Best to use POST method regardless.

About This Series

This is the third post in a series of how to extract data from an ArcGIS Server map services to a local shapefile FC or Geodatabase FC. The first post “Extract a Map Service Layer to Shapefile using Python” focused on the basics: query, save as JSON, convert to Shapefile. The second post, ”Map Service to Shapefile with Python Part 2 – Iteration” dealt with larger datasets, when the results were bigger than a single query would allow.