Step 1 (took 30 minutes to run) python process_ghcnd_by_stn.py stnlist_all_2021Jan29.txt 1940 2020 This works fine, even with the ISD stations in the station list. It quickly identifies that the station is unavailable and goes to the next one. No error/warning message is output, which I think is fine. Note that processing time should be somewhat similar regardless of the specified start/end years. While there will be some differences in the processing time, the files being downloaded from NCEI contain the entire period of record for each station. So specifying a smaller subset of years has no effect on download time. The ftp server load and other similar factors will also be relevant, making the actual processing time variable. One thing to note... When this script finishes, check the output to be sure that you were able to download ALL relevant files. The script is configured to gracefully handle an issue with any particular station being missing or unavailable. But for some reason, once it fails it seems to fail every subsequent one. Step 2 (just under 8 hours to run) python get_all_ish.py stnlist_isd_2021Jan29.txt 1940 2020 This will work even with the GHCND stations in the list. But it wastes a *LOT* (several hours) of time trying to download each of those missing GHCND files. Thus, I strongly recommend editing the stnlist file to only contain the ISD stations. The listed time of 8 hours was when I ran it with only the isd stations in the list. My first attempt had the ghcn stations at the top of the list, and I noted after a couple hours that it was only partway through that section of the list. I don't remember exactly how far, but it was still in the Canadian ones and hadn't even reached the U.S. yet. So I'm guessing it would have taken 6-12 additional hours had I not stopped it and restarted with an edited list. Step 3 (just over 32 hours to run) run GLSHFS to add all of the data 1. initial setup 2. Edit the GLSHFS config file: AddStationData = YES BuildSubbasinMet = YES UpdateHistorical = NO 3. run the procedure Step 4 (took 1h 12m) Run GLSHFS (without adding new data) for the modeling phase 1. Edit the GLSHFS config file: AddStationData = NO BuildSubbasinMet = NO UpdateHistorical = YES Note that steps 3 & 4 can be combined by simply setting all 3 of those switches to YES. I recommend doing the big historical set in separate steps only to avoid wasted time if there is an error in the station data update. Step 5 Build station met for the "operational" update, to simulate USACE process 1. Delete all of the met_*.csv files in the STN directory 2. python process_ghcnd_by_stn.py stnlist_all_2021Jan29.txt 2020 2021 (took 18 minutes) 3. python get_all_ish.py stnlist_isd_2021Jan29.txt 2020 2021 (took under an hour, but not exactly sure) 4. Edit the GLSHFS config file: AddStationData = YES BuildSubbasinMet = YES UpdateHistorical = YES 5. run GLSHFS (took 2h 50m) Step 5 (take 2) Hmm... That step 5 above didn't make forecasts. USACE wants to know how long it will take to do EVERYTHING. So let's do step 5 again, but this time we will actually do the forecast part, too. 1. Delete all of the met_*.csv files in the STN directory 2. python process_ghcnd_by_stn.py stnlist_all_2021Jan29.txt 2020 2021 (took 26 minutes) 3. python get_all_ish.py stnlist_isd_2021Jan29.txt 2020 2021 (took 14 minutes) 4. Edit the GLSHFS config file: AddStationData = YES BuildSubbasinMet = YES UpdateHistorical = YES RunForecasts = YES MakeSummaryFiles = YES OutputMetDetails = [] 5. run GLSHFS (took 3h 17m)