Project Development Best Practices
December, 2023
Download the PDF here
Introduction
Arc Hydro is a data model, toolset, and collection of workflows developed over the years to support specific GIS implementations in water resources. The initial implementation of Arc Hydro was in 2002, beginning with the data model, the Arc Hydro book published by Esri Press, and an initial set of about 30 tools. Since then, Arc Hydro has been used in many projects and in the process new tools and workflows have been developed. Arc Hydro tools now number in the 300s and continue to be expanded based on work on specific implementations.
This document captures what works well when implementing Arc Hydro projects. It is based on many years of implementing ArcGIS and Arc Hydro projects across the world, support for Arc Hydro users, training, and specific user feedback (always welcome). There are other recommendations that could/should be made, and this document will be expanded in the future to include them as they become apparent.
This document is not focused on a specific version of ArcGIS or Arc Hydro - it is a set of generic recommendations that transcend specific versions. Most of these recommendations will work well for any ArcGIS project, not just the ones based on Arc Hydro.
Document history
Table 1. Document Revision History
| Version | Description | Date |
|---|---|---|
| 1 | First Version (DD) | 02/2013 |
| 2 | Minor revisions, new "look" (DD) | 01/2019 |
| 2.1 | Minor revisions, ArcGIS Pro reference (DD) | 08/2019 |
| 2.2 | Minor revisions (DD) | 11/2019 |
| 2.3 | Minor revisions (DD, GLO, CD) | 12/2023 |
The Arc Hydro mantra
Here is some basic information about Arc Hydro to keep in mind:
Arc Hydro is a "system" of tools and data structures that work in unison to provide rich and effective experience for GIS users in the water resources community.
If you are using Arc Hydro, you will most likely be doing ANALYSES - there are important considerations for GIS implementation when doing analyses that are different than using GIS just for mapping. Respect that.
When using the tools, you are building a "system", not just using a "bunch" of independent tools to produce a "bunch" of independent data.
- Of course, there are plenty of Arc Hydro tools that can be used independently of the "system". Use them as such and enjoy.
When starting an Arc Hydro project, think ahead of the system you will be building. Plan ahead. Organize, then execute.
Keep it simple - Ockham 's razor is alive and well!
Analytical system, analytical system, analytical system, ... Always consider how what you are doing fits into the system you are developing. Understand what was done before and what comes after the step you are working on.
Arc Hydro Tools installation
When installing Arc Hydro tools:
Arc Hydro's [Community] home page will have a pointer to the latest released versions of Arc Hydro tools and specific comments related to the core version of the software (e.g. any required patches or service packs).
Arc Hydro tool setup can change often. That does not mean that you need to upgrade each time a new setup is released - do it only if the changes are relevant to you (e.g. a bug fix or a new functionality that you would be using). "If it ain't broke, don't fix it".
Always uninstall before installing.
When upgrading to new release of ArcGIS or service pack (SP), always check if there is a new release of Arc Hydro tools. Do not assume that they are upward (or downward) compatible.
- This is mostly between major core version updates, but sometimes SPs will need a separate version. Check the "Arc Hydro Installation - Versions and Documentation" on Arc Hydro Community page for clarification of which Arc Hydro tools are compatible with which core version/SP combination.
Data organization
When organizing Arc Hydro data:
Use file geodatabase or SDE for vector data.
Keep rasters outside of the geodatabase.
Do NOT use unprojected GIS data for analysis.
Keep all the vector layers used in spatial operations and analytics in a single feature dataset. This recommendation has several corollaries (for analytical layers):
Do not use shape files.
Do not use data from different geodatabases.
Import any new data into the single feature dataset used throughout the project.
Use the same spatial reference (projection) for raster and vector analytical data.
Select the appropriate projection based on the analysis being performed (usually equal area or equal distance - something that minimizes distortions of the data relevant to the analysis).
Layers used for mapping ONLY, can be in different feature datasets or at the geodatabase level but keep the same spatial reference as for analytical layers if possible.
- Exceptions are display layers that will be published for web display - in that case use Web Mercator or similar projection.
Organize all the data for a project into a single "parent" folder if possible.
Do NOT use the data over a network drive if possible. Instead, copy the data to the local machine, perform the analysis locally, and then copy the outputs back to the central location.
- The exception is if you have a specialized network drive with specialized network connection hardware.
Use internal hard drive (HD) if possible (not USB).
- Use internal solid-state drive (SSD) if possible.
If using default Arc Hydro geodatabase/feature dataset creation capabilities, make sure that the first function that will create the geodatabase/feature dataset is picking up the right input spatial reference (either from the input dataset or map data frame).
Once the initial project database is assembled, carefully manage HydroIDs if any of the input layers have them (they might have to be recalculated). Make sure you adjust the HydroID dispenser field (LASTID) in the APUNIQUEID table.
Set vertical units for the digital elevation model (DEM) if not the same as the horizontal units.
- It is recommended to keep the horizontal and vertical units the same.
Naming conventions
The following naming conventions should be followed:
Keep the names of directories, geodatabases, layers, and attributes short.
- Do not have too many nested subdirectories containing your project folder.
Do not use spaces or special characters in any of the names. An underscore (_) is the only exception.
Start names always with a letter, never with a number.
Do not use reserved keywords for names. This includes SQL commands and ArcGIS commands (e.g. WHERE, SUM, ...).
Develop a "standard" layer nomenclature and apply it consistently (e.g. start all flow direction rasters with "fdr" followed by descriptor of what flow direction is based on - e.g. "fil" for filled DEM, so "fdrfil" is a flow direction derived from a filled DEM).
If using more than one specific stream threshold in the project, include the threshold in the name of the layer (e.g. str50k to indicate a stream grid based on a threshold of 50,000 cells).
- Recommended even when a single threshold is used in the project.
Keep raster names to no more than 13 characters (although this is not required for non Esri grid formats - e.g. .tif).
Use input rasters that have the same cell size, extent, and snap layout as the underlying analytical raster (e.g. DEM and/or flow direction).
Initial database/mxd/aprx organization
Standard Arc Hydro database/project organization consists of (recommended, not required):
Project folder that will contain all the project data and mxd/aprx. This excludes base data (data not used in analytical operations) that can be anywhere.
A "Layers" subfolder. This folder is used to store all the raster data.
DEM is usually the starting point of the analyses. Copy the clipped DEM raster into this folder. Clipping should be based on the relevant project area of interest. Apply projection if needed. Apply vertical unit adjustments if needed.
If other rasters are used as inputs, copy them into this folder as well. Apply projection if needed.
A project geodatabase.
- A "Layers" feature dataset is created within the geodatabase using DEM's spatial reference. Input vector data used in the project are imported into this feature dataset.
A project mxd/aprx. Name the mxd the same as the geodatabase (e.g. if you have aaa.gdb, call your mxd aaa.mxd or your aprx aaa.aprx).
Follow this workflow if starting from DEM and using Arc Hydro tools to generate project geodatabase (for ArcMap tools):
Open a new ArcMap. Add the project DEM to it first, BEFORE any other layer (or base map). This will set the spatial reference of the data frame to the spatial reference of the DEM.
Save the project using the same name you want to give to the geodatabase. Arc Hydro tools will create the geodatabase and the "Layers" feature dataset when needed - you do not have to do it yourself. This will happen when the first Arc Hydro function that generates a vector output is run.
If using ArcGIS Pro version of Arc Hydro tools, review document "Arc Hydro - ArcGIS Pro Project Startup Best Practices" on different alternatives in Pro for project startup that will establish well-defined Arc Hydro project structure.
General Items
Following these guidelines will improve your Arc Hydro implementation experience:
Leverage Arc Hydro's [Community] home page to interact with broader Arc Hydro user's community (including Esri's Arc Hydro development and maintenance team).
If still using ArcMap:
Perform Arc Hydro geoprocessing analyses through ArcMap (not ArcCatalog).
If you need to use ArcCatalog, use it from within ArcMap if possible instead of starting a separate instance of ArcCatalog.
Start a new Arc Hydro project only from a newly opened ArcMap session (that is, start a new instance of ArcMap when starting a new Arc Hydro project).
Background processing will in general allow processing of larger datasets, but not necessarily any faster. And it can cause some functions to malfunction. Test the function in the background with a smaller dataset first and if it completes, run it on the larger dataset.
- You need to have 64-bit geoprocessing support installed for background processing to have any effect.
Save your project often.
Save after each major subtask.
Do not have multiple ArcMap/Pro projects open at the same time.
When functions that generate new feature classes have been executed (creating domains in particular), it is good to save and close the project, and then reopen the project.
Do not change project name (e.g. save as ...).
If you are doing a lot of geoprocessing and do not need the logs, wipe them out often or turn logging off (suggested only when working on datasets that were previously tested).
Geoprocessing in general and Arc Hydro in particular generates a lot of temporary files that are stored in a %temp% system directory. It is good to wipe that directory clean from time to time (delete all the files in there that you can after closing all the apps).
Do not use default ArcGIS geoprocessing target location to store your results - be explicit where results are directed (your project directory and geodatabase).
Be aware that some of the Arc Hydro functions need higher versions of licensing.
To improve performance of geoprocessing tools that generate many intermediate layers:
Turn off all the layers in the map (no data are displayed).
Go to ArcMap/Pro Options -> General and uncheck option "Make newly added layers visible by default".
Debugging and workarounds
Things will go wrong.
They will not go wrong often if you are following the suggestions above.
There are several types of problems you will face, and each will require a different approach to identifying and fixing or working around.
Hardware problems. Lack of system memory and/or hard drive space will often either generate ArcMap crash or a generic error that will be difficult to interpret. If something used to work, but does not now, check if available resources have changed (HD space or available memory).
Some raster functions fail on large integer rasters but work on smaller. In ArcMap (not available in Pro) try to increase maximum number of entries a raster attribute table can have. Note that while this can increase stability of the system, it might also somewhat degrade its performance.
Use "ArcMap Advanced Settings Utility" (you need to be administrator to configure some settings) and increase "Attribute Limit" under "Raster" tab from default 65,536 to 1,000,000. This application can be found in the "Utilities" folder under ArcMap installation directory (typically for ArcGIS 10.x it is C:\Program Files (x86)\ArcGIS\Desktop10.x\Utilities).
If you do not have administrator access, you can change this setting in ArcMap options. Go to "Customize -> ArcMap Options... -> Raster -> Raster Dataset" and modify "Maximum number of unique values to render" value. Note that "Raster" tab in the options, has five subtabs. Make sure you scroll to the left if you do not see the "Raster Dataset" subtab initially.
Use "Copy Raster" to copy a raster dataset that might be causing problems. Then use this copy in your work. This might be needed with some rasters not developed using Esri software.
Data problems - your data are bad - do not expect tools to work. Fix the data.
Make sure you do not have an active selection before executing the tool if you wanted to process everything.
Data problems - your data are too big. Divide and conquer - split the dataset into smaller "chunks" and process in parts. This might also speed up the overall processing time as processing times are non-linear (especially in the raster domain).
- In many water resources applications, watersheds are "natural" chunks and a good starting point when considering how to chunk up the data. In the USA, WBD/HUCs are a great starting point for AOI definition.
Procedural problems - those will be minimized if you followed the best practices presented here (e.g., projection mishaps will be eliminated) and are following well-established Arc Hydro processing workflows.
Tool bugs - yes, there might be some, especially after new core software releases or major service packs. If you identify any, please let us know via Community and we will fix them (often you will have to provide the data if we cannot reproduce the bug on our test datasets).
"Gremlins". Things that used to work in an existing project suddenly do not. This is usually related to file locking issues. Try to fix by applying fixes in the following order:
Make sure you have enough HD space and working memory (if you are not running 64-bit processing/background, you are limited to how much memory is accessible for processing within ArcMap session).
Make sure there is only one instance of ArcMap/Pro open (this also includes ArcCatalog).
Make sure the source and target locations, and layer tags are all properly set.
Close ArcMap/Pro and reopen.
Close ArcMap/Pro and all other apps, delete content of %temp% directory, reopen.
Close all apps, reboot, reopen.
Delete the mxd/aprx. Start the new project, add only the data needed in the analysis, and configure the layers/inputs as needed by the tools.